The CI contract: 40 tests across six groups

40 tests 14 leakage-spine guards, 5 ONNX parity, 3 predict-schema, 7 evaluation gates, 1 target bound, 10 version-contract checks. Run them all with make ml-test.

The ML layer’s 40 tests are load-bearing, not box-ticking. The leakage spine is the only thing standing between a model that looks good on paper and one that silently inflates every offline number by memorising the season. The ONNX parity gate is the only thing that ensures the browser scores what the Python booster trained. Every test has a unique idea; none is decorative.

Run all ML tests

make ml-test

The 40 tests, grouped

Group	Source file	Count	What it guarantees
Leakage Spine	test_features.py	28	No target, skill, or season column enters X; forward-window SQL audit (including its own coverage, and self-join horizons); aggregation-scope audit, which rejects a GROUP BY that does not pin a lap unless the model declares it in schema.yml with a reason, and is itself falsified against both known shapes; holdout purity; MAX+1-derived split; targets bounded by the clip their own column carries, non-null, and never also features at any horizon.
ONNX Parity	test_onnx_parity.py	5	Each booster round-trips to ONNX within atol=1e-5, including a NaN-bearing sample (the ~47% null-prior laps).
Predict Schema	test_predict.py	3	Scored predictions parquet carries the declared 19-column schema (17 + the p10/p90 stint-life band); Arrow-validated.
Evaluation Gates	test_evaluate.py	19	Every model beats its per-cohort baseline; calibration coverage computed; cohorts surfaced not dropped; every model carries an attainable ceiling and every beats-baseline claim an interval.
CRPS	test_crps.py	5	The proper-score arithmetic, on synthetic data: a collapsed trio integrates exactly to MAE, the mcb - dsc + unc decomposition reconstructs CRPS on imperfectly-calibrated predictions, unc matches the unconditional pinball loss at each alpha, and a single quantile level is refused.
Targets	test_targets.py	3	Stint-life target is synthesised without leaking stint_length_laps; the censoring flag rides in metadata and never becomes a feature; AFT bounds encode censoring as a point vs a half-line.
Survival	test_survival.py	18	The AFT contract itself: the +1 shift keeps zero-life stints off log(0), margins round-trip to laps, quantiles bracket the median and widen with scale, censored rows are not punished for over-prediction, and C-index ranks. aft_params refuses a non-AFT booster.
Version Contract	test_manifest_contract.py	18	The manifest names a version whose artefacts exist, declares the input width the boosters actually take, and matches the copy the browser loads; every beats_baseline on the card carries an interval, and that gate is proven to fire.
Attainable Ceilings	test_ceiling.py	19	The between-stint variance estimator recovers a known ICC where the naive one inflates it; rolling-window overlap is detected and thinned; the in-sample stint oracle really is the optimum over stint-constant predictors; clustered intervals are wider than lap-grain ones.
Within-stint Attribution	test_attribution.py	33	The flatten is a pure within-stint operation; a planted per-lap driver and a planted stint-level one separate under it where a drop cannot tell them apart; the causal summary over laps 1..t provably never reads a later lap; a planted look-ahead result is named unreachable.
Fit Parity	test_fit_parity.py	24	The search, the evaluation refit and the production refit fit the same model: the quantile trio’s IPW survival weights and the AFT censoring flag reach every fold of every path, sliced to that fold’s own rows; a quantile fit offered no weights raises instead of defaulting; and a refit that would replace a version fitted on a different target column is refused.
Search Space	test_search_space.py	14	The hyperparameter space is declared once and _suggest builds from it, so no suggestion can leave the bounds the pin detector reads; boundary_params names a best-params set that stopped on an edge, and is proven to fire at each ceiling, at each floor, and to stay silent in the interior.
Total		189

Go deeper

Leakage Spine

The 14 guards in detail one idea each, with the assertion and why it matters.

Parity & Schema

The 5 ONNX parity + 3 predict-schema + 1 target tests the mechanical contracts.

Evaluation Gates

The 7 evaluation gates beats-baseline, calibration computed, cohorts surfaced.

Version Contract

The 10 checks that the manifest names the version it shipped, declares the input width the boosters actually take, and matches the copy the browser loads.

Validation

The methodology behind the metrics these gates enforce.