Distribution performance and regression preservation

Numerical comparisons before timings

All input cases are verified across implementations before any timing phase begins. Seventeen evenly spaced output values plus min/max use the distribution fixture tolerance: 1e-300 + 1e-12*abs(reference), or 1e-15 + 1e-12*abs(reference) for quantiles. Sum and sum-of-squares consume all outputs, with tolerance 1e-9 + 1e-10*abs(reference) to allow different accumulation orders. Mean/variance outputs are checked directly. R is the benchmark reference; the separate R/SciPy/mpmath record adjudicates reference differences and tests extreme/boundary values more broadly.

Native comparisons must pass. SciPy peer discrepancies are retained and labeled; they do not silently relax the native gate. Every timed output summary is checked again. This benchmark does not compare every individual element of the large vectors; the complete-vector aggregates and sampled values complement the strict small reference/identity suite.

Recorded: 2026-10-12T15:18:59.861643+00:00; Apple M5 Ultra, macOS-27.0.1-arm64-arm-64bit-Mach-O. 27 input cases; all native/R comparisons passed. SciPy exceeded the same tolerance for: chisquare_cdf (1,000), chisquare_cdf (100,000), chisquare_cdf (1,000,000).

Median milliseconds per full call, 5 measurements after one warm-up:

Whole-process peak RSS across these workers reached 268.1 MiB; per-worker memory, load/startup times and every timing sample are retained in the raw report.

Timing boundaries and interpretation

Each worker loads/parses inputs once, performs one untimed warm-up, and measures five repetitions. Distribution calls allocate and populate the complete output vector. The PHP scalar loop constructs a distribution and crosses the PHP/native boundary for each value; batch calls construct it once and stream values natively. The internal Rust loop uses the same validation and an allocated result vector. Descriptive mean/variance has one PHP native call; there is no invented scalar-loop equivalent for those operations.

JSON I/O, startup and output summaries are outside timed calls. PHP input/output conversion and native validation are included. R/SciPy start with native numeric vectors. Runtime reclamation/garbage collection of prior results contributes to the observed API cost. Peak RSS is the whole worker process, including imports, JSON parsing, inputs, outputs and summaries, not isolated kernel memory. Process elapsed time and input loading time are recorded separately.

R’s timer has millisecond resolution. Small distribution cases batch 100 calls; mean/variance batch enough calls to process at least ten million input values per measurement, with a minimum of ten calls. Per-call times divide by the batch count. All raw timings must be positive. Thread environment variables are fixed to one for OpenBLAS, OpenMP, MKL, Accelerate and BLIS; they do not make different numerical implementations identical. PHP scalar/batch worker order alternates by case. Machine load, timer precision and GC make these measurements evidence for these workloads rather than universal speed claims.

Seven-model preservation

The preservation report compares the current extension with a rebuilt pre-v0.2 baseline at 6342562, using the same PHP 8.5 headers/compiler. Both receive identical complete designs with intercept insertion disabled where supported. It exercises all seven existing fitters. Reference tolerance is 1e-7 + 1e-6*abs(reference) for the compared inference/scalar outputs, group effects and ten sampled observation outputs. Fits materialize their complete public results before summarization. This is separate from fitting all independent R fixtures in the release gate.

Recorded: 2026-10-12T15:18:25.373311+00:00; baseline revision 6342562. All seven model comparisons passed.

This check does not establish bitwise agreement for every observation output or a performance bound on every future workload. Retained regression paths are also covered by the existing Rust/PHP reference and intercept/offset suites.

Reproduction

python3 -m venv .context/v02/python
.context/v02/python/bin/pip install -r tests/benchmarks/distribution-requirements.txt
# R with jsonlite is a development-only reference dependency.
cargo build -p phit --release --locked
cargo build -p stats-core --release --locked --example distribution_bench
PHP=/path/to/matching/php .context/v02/python/bin/python tests/benchmarks/run-distributions.py
python3 scripts/distribution-docs.py

The runner accepts --sizes, --operations, --repeats, --output and --resume. A partial report is marked complete: false; resume requires matching binary hashes, seed, repetitions, thread settings and input hashes. It keeps completed verification/timings and recomputes any changed input. Use a new output file for a different build. Fixed seed 20261012 and saved NumPy/Python versions specify input generation; SHA-256 records the exact JSON values actually used.

Rebuild the preservation baseline in an isolated source/target directory, using the same PHP/Clang environment, then run:

mkdir -p .context/v02/baseline-src
git archive 6342562 | tar -x -C .context/v02/baseline-src
CARGO_TARGET_DIR="$PWD/.context/v02/baseline-target" cargo build \
  --manifest-path .context/v02/baseline-src/Cargo.toml -p phit --release --locked
# macOS; use libphit.so on Linux.
PHP=/path/to/matching/php .context/v02/python/bin/python tests/benchmarks/run-v01.py \
  --baseline .context/v02/baseline-target/release/libphit.dylib \
  --baseline-revision 6342562 --output docs/benchmarks/v02-preservation-results.json
python3 scripts/distribution-docs.py

The worker uses reflected function parameters, so the older module name and API remain loadable without importing current stubs into the baseline runtime. No benchmark changes a user’s PHP configuration or needs network/model services.