The CI contract: every ML test, grouped

The ML test suite by group: leakage spine, ONNX parity, predict schema, evaluation gates, CRPS, targets, survival, version contract, attainable ceilings, attribution, fit parity, search space and the power-law tyre curve. Run them all with make ml-test.

The ML layer’s tests are load-bearing, not box-ticking. The leakage spine is the only thing standing between a model that looks good on paper and one that silently inflates every offline number by memorising the season. The ONNX parity gate is the only thing that ensures the browser scores what the Python booster trained. Every test has a unique idea; none is decorative.

Run all ML tests

make ml-test

​

The tests, grouped

The table is generated from the live pytest collection, so its counts are always current.

Group	Source file	Count	What it guarantees
Leakage Spine	test_features.py	34	No target, skill, or season column enters X; forward-window SQL audit (including its own coverage, and self-join horizons); aggregation-scope audit, which rejects a GROUP BY that does not pin a lap unless the model declares it in schema.yml with a reason, and is itself falsified against both known shapes; holdout purity; MAX+1-derived split; targets bounded by the clip their own column carries, non-null, and never also features at any horizon.
ONNX Parity	test_onnx_parity.py	5	Each booster round-trips to ONNX within atol=1e-5, including a NaN-bearing sample (the ~47% null-prior laps).
Predict Schema	test_predict.py	3	Scored predictions parquet carries the declared 19-column schema (17 + the p10/p90 stint-life band); Arrow-validated.
Evaluation Gates	test_evaluate.py	19	Every model beats its per-cohort baseline; calibration coverage computed; cohorts surfaced not dropped; every model carries an attainable ceiling and every beats-baseline claim an interval.
CRPS	test_crps.py	5	The proper-score arithmetic, on synthetic data: a collapsed trio integrates exactly to MAE, the mcb - dsc + unc decomposition reconstructs CRPS on imperfectly-calibrated predictions, unc matches the unconditional pinball loss at each alpha, and a single quantile level is refused.
Targets	test_targets.py	3	Stint-life target is synthesised without leaking stint_length_laps; the censoring flag rides in metadata and never becomes a feature; AFT bounds encode censoring as a point vs a half-line.
Survival	test_survival.py	36	The AFT contract itself: the +1 shift keeps zero-life stints off log(0), margins round-trip to laps, quantiles bracket the median and widen with scale, censored rows are not punished for over-prediction, and C-index ranks. aft_params refuses a non-AFT booster. The survival metrics are pinned on synthetic data: D-calibration passes a correctly specified fit and fails a biased one, IPCW Brier and time-dependent AUC reward the better model, and the copula-graphic estimator reduces to Kaplan-Meier at tau = 0.
Version Contract	test_manifest_contract.py	30	The manifest names a version whose artefacts exist, declares the input width the boosters actually take, and matches the copy the browser loads, ONNX hashes included; the survival output block reconstructs the booster; every beats_baseline on the card carries an interval; no ML-contract mart reads the driver-isolation ratings; and the interval and lineage gates are each proven to fire.
Attainable Ceilings	test_ceiling.py	19	The between-stint variance estimator recovers a known ICC where the naive one inflates it; rolling-window overlap is detected and thinned; the in-sample stint oracle really is the optimum over stint-constant predictors; clustered intervals are wider than lap-grain ones.
Within-stint Attribution	test_attribution.py	33	The flatten is a pure within-stint operation; a planted per-lap driver and a planted stint-level one separate under it where a drop cannot tell them apart; the causal summary over laps 1..t provably never reads a later lap; a planted look-ahead result is named unreachable.
Fit Parity	test_fit_parity.py	31	The search, the evaluation refit and the production refit fit the same model: the quantile trio’s IPW survival weights and the AFT censoring flag reach every fold of every path, sliced to that fold’s own rows; a quantile fit offered no weights raises instead of defaulting; and a refit that would replace a version fitted on a different target column is refused.
Search Space	test_search_space.py	15	The hyperparameter space is declared once and _suggest builds from it, so no suggestion can leave the bounds the pin detector reads; boundary_params names a best-params set that stopped on an edge, and is proven to fire at each ceiling, at each floor, and to stay silent in the interior.
Power-law Tyre Curve	test_powerlaw_export.py	37	Neither the feature allowlist nor the training frame carries a cliff-seed, expected_* or compound-net column, and the denylist is proven to catch each one; the shape model never sees the level-only features; the per-cell fit matches dbt; the curve is monotone and zero at its anchor; the two ONNX files stay under 500 KB together and match XGBoost to 1e-5 on the test fold.
Total		270

​

Go deeper

Leakage Spine

The leakage guards in detail one idea each, with the assertion and why it matters.

Parity & Schema

The ONNX parity, predict-schema, target and version-contract tests: the mechanical contracts.

Evaluation Gates

Beats-baseline, calibration computed, cohorts surfaced, ceilings and intervals.

Version Contract

The checks that the manifest names the version it shipped, declares the input width each booster actually takes, and matches the copy the browser loads.

Validation

The methodology behind the metrics these gates enforce.