Benchmarks and verification
The same random-intercept ML model in R, Python and PHP
The math walkthrough derives the marginal Gaussian likelihood, group whitening, variance profiling and shrunken conditional group estimates. All runtimes fit the same fixed columns, retained observations and grouping IDs.
from pathlib import Path
import sys
import pandas as pd
ROOT=next(p for p in [Path.cwd(),*Path.cwd().parents] if (p/'Cargo.toml').exists())
sys.path.insert(0,str(ROOT/'docs/verifications'))
from compare_lmm import cases,compare_case
results=[compare_case(c) for c in cases()]
first=results[0]
print(first['versions'])
rows=[]
for implementation in ['R','Python','php-stats']:
fit=first[implementation]
for j,label in enumerate(['intercept','study hours','prior score']):
rows.append(dict(implementation=implementation,term=label,estimate=fit['coefficients'][j],SE=fit['std_errors'][j],z=fit['z_statistics'][j],p=fit['p_values'][j],lower95=fit['conf_int'][j][0],upper95=fit['conf_int'][j][1]))
display(pd.DataFrame(rows))
display(pd.DataFrame([dict(case=r['name'],Python_backend=r['python_diagnostics']['backend'],
max_absolute_difference_vs_R=max(e for values in r['max_absolute_errors_vs_R'].values() for e in values.values()),
PHP_boundary=r['php-stats']['boundary'],missing_warnings=len(r['warnings'])) for r in results])){'R': 'R version 4.6.1 (2026-06-24)', 'lme4': '2.0.6', 'statsmodels': '0.15.0', 'PHP': '8.5.10', 'Python': '3.14.7'}
| implementation | term | estimate | SE | z | p | lower95 | upper95 | |
|---|---|---|---|---|---|---|---|---|
| 0 | R | intercept | 46.655187 | 2.204067 | 21.167768 | 1.893116e-99 | 42.335294 | 50.975079 |
| 1 | R | study hours | 1.613908 | 0.160247 | 10.071362 | 7.394689e-24 | 1.299829 | 1.927987 |
| 2 | R | prior score | 0.245431 | 0.041713 | 5.883790 | 4.009775e-09 | 0.163675 | 0.327187 |
| 3 | Python | intercept | 46.655187 | 2.204067 | 21.167768 | 1.893113e-99 | 42.335294 | 50.975079 |
| 4 | Python | study hours | 1.613908 | 0.160247 | 10.071362 | 7.394690e-24 | 1.299829 | 1.927987 |
| 5 | Python | prior score | 0.245431 | 0.041713 | 5.883790 | 4.009775e-09 | 0.163675 | 0.327187 |
| 6 | php-stats | intercept | 46.655187 | 2.204067 | 21.167768 | 1.893113e-99 | 42.335294 | 50.975079 |
| 7 | php-stats | study hours | 1.613908 | 0.160247 | 10.071362 | 7.394690e-24 | 1.299829 | 1.927987 |
| 8 | php-stats | prior score | 0.245431 | 0.041713 | 5.883790 | 4.009775e-09 | 0.163675 | 0.327187 |
| case | Python_backend | max_absolute_difference_vs_R | PHP_boundary | missing_warnings | |
|---|---|---|---|---|---|
| 0 | simulated_schools | statsmodels MixedLM (ML) | 4.783827e-07 | False | 0 |
| 1 | simple | statsmodels MixedLM (ML) | 1.490081e-06 | False | 0 |
| 2 | no_intercept | statsmodels MixedLM (ML) | 1.641136e-04 | False | 0 |
| 3 | intercept_only | statsmodels MixedLM (ML) | 4.142385e-07 | False | 0 |
| 4 | singleton_groups | statsmodels MixedLM (ML) | 5.790743e-07 | False | 0 |
| 5 | boundary | NumPy exact zero-random-variance limit | 7.105427e-15 | True | 0 |
| 6 | listwise_missing | statsmodels MixedLM (ML) | 1.126244e-06 | False | 1 |
Comparisons include every fitted value/residual, fixed-effect inference, group effects, variance components, ICC and marginal likelihood at tolerance 1e-7 + 1e-6 × abs(reference). Group labels, deletion positions and counts are checked exactly. Optimizer evaluation counts are not expected to match.
Equivalent model calls
library(lme4)
students <- read.csv('docs/data/schools.csv')
fit <- lmer(score ~ study_hours + prior_score + (1|school),
data=students, REML=FALSE,
control=lmerControl(optimizer='bobyqa',optCtrl=list(rhoend=1e-10)))
fixef(fit)
vcov(fit)
ranef(fit)import statsmodels.api as sm
students = pd.read_csv('docs/data/schools.csv')
x = sm.add_constant(students[['study_hours','prior_score']])
fit = sm.MixedLM(students.score, x, groups=students.school).fit(
reml=False, method='bfgs', gtol=1e-10, maxiter=1000)
fit.fe_params, fit.scale, fit.cov_re$x = $y = $groups = [];
foreach ($students as $student) {
$x[] = [1.0, $student['study_hours'], $student['prior_score']];
$y[] = $student['score'];
$groups[] = $student['school'];
}
$result = stats_lmm(x: $x, y: $y, groups: $groups);These PHP records are already numeric; examples/lmm.php supplies the complete CSV conversion. No formulas or implicit intercepts are interpreted by the native call. The comparison runner gives R the exact design columns with formula y ~ 0 + x0 + ... + (1|group).
Conventions that must match
ML is explicit: R and statsmodels otherwise support/default to REML, which fits a different variance criterion. Likelihood here integrates over group effects. Fixed-effect standard errors use plug-in GLS covariance (X’ V^-1 X)^-1, matching lme4. statsmodels’ reported joint observed-Hessian covariance can differ, so the Python runner explicitly recalculates the declared GLS convention using its fitted variance components; it does not claim its unmodified bse_fe matches every case. Normal z-based Wald intervals are constructed explicitly for every reference.
Our df_residual=n-p is informational. R mixed-model df methods may include the two variance parameters; no finite-sample t degrees of freedom are claimed here. AIC counts p+2 parameters in every case, including a zero variance boundary.
The exact tau²=0 fixture is checked against lme4 and PHP, with a separately labeled NumPy OLS-limit reference. It is not represented as a converged statsmodels mixed fit. At this boundary, variance is RSS/n and standard errors differ from OLS’s unbiased-scale inference. This explicit check avoids treating an approximate positive-variance optimizer solution as proof of exact boundary behavior.
keys=['residual_variance','random_intercept_variance','icc','log_likelihood','aic']
display(pd.DataFrame([dict(implementation=name,**{k:first[name][k] for k in keys}) for name in ['R','Python','php-stats']]))
print('Python fit warnings:', first['python_diagnostics']['warnings'])
missing=results[-1]
print('Dropped original rows:',missing['php-stats']['dropped_rows'])
print('Missing-data warning:',missing['warnings'])
print('Schools and effects:',list(zip(first['php-stats']['group_labels'],first['php-stats']['random_effects'])))| implementation | residual_variance | random_intercept_variance | icc | log_likelihood | aic | |
|---|---|---|---|---|---|---|
| 0 | R | 25.355429 | 18.105139 | 0.416588 | -775.374199 | 1560.748398 |
| 1 | Python | 25.355429 | 18.105139 | 0.416588 | -775.374199 | 1560.748398 |
| 2 | php-stats | 25.355429 | 18.105139 | 0.416588 | -775.374199 | 1560.748398 |
Python fit warnings: []
Dropped original rows: [1, 4, 8]
Missing-data warning: [{'severity': 2, 'message': 'stats_lmm(): stats_lmm dropped 3 of 250 observations with missing values (listwise deletion)'}]
Schools and effects: [(107, 0.13413303204091775), (114, 1.6086975147685796), (121, -1.7846684515271147), (128, -5.564364724188796), (135, 0.24907018967277064), (142, 2.2564389505860314), (149, 10.331238035927292), (156, -5.104699176036904), (163, 2.102747547050482), (170, 0.5690811113270031), (177, 0.07903267727850147), (184, -4.87670670689903)]
See the verification record for rejection tests, finite-search limitations and reproduction instructions. A converged numerical fit does not establish that the model assumptions or small-group Wald inference are adequate for an application.
From phpstats/phit@76b22f0.