> ## Documentation Index
> Fetch the complete documentation index at: https://offthepace.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# The CI contract: 40 tests across six groups
> 40 tests 14 leakage-spine guards, 5 ONNX parity, 3 predict-schema, 7 evaluation gates, 1 target bound, 10 version-contract checks. Run them all with make ml-test.
The ML layer's 40 tests are load-bearing, not box-ticking. The leakage spine is the only thing standing between a model that looks good on paper and one that silently inflates every offline number by memorising the season. The ONNX parity gate is the only thing that ensures the browser scores what the Python booster trained. Every test has a unique idea; none is decorative.
```bash Run all ML tests theme={null}
make ml-test
```
```bash Run a specific group theme={null}
./.venv/bin/python -m pytest ml/tests/test_features.py -v # leakage spine
./.venv/bin/python -m pytest ml/tests/test_onnx_parity.py -v # ONNX parity
./.venv/bin/python -m pytest ml/tests/test_predict.py -v # predict schema
./.venv/bin/python -m pytest ml/tests/test_evaluate.py -v # evaluation gates
./.venv/bin/python -m pytest ml/tests/test_targets.py -v # targets
```
## The 40 tests, grouped
| Group | Source file | Count | What it guarantees |
| ---------------------------- | --------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Leakage Spine** | `test_features.py` | 28 | No target, skill, or season column enters `X`; forward-window SQL audit (including its own coverage, and self-join horizons); aggregation-scope audit, which rejects a `GROUP BY` that does not pin a lap unless the model declares it in `schema.yml` with a reason, and is itself falsified against both known shapes; holdout purity; `MAX+1`-derived split; targets bounded by the clip their own column carries, non-null, and never also features at any horizon. |
| **ONNX Parity** | `test_onnx_parity.py` | 5 | Each booster round-trips to ONNX within `atol=1e-5`, including a NaN-bearing sample (the \~47% null-prior laps). |
| **Predict Schema** | `test_predict.py` | 3 | Scored predictions parquet carries the declared 19-column schema (17 + the p10/p90 stint-life band); Arrow-validated. |
| **Evaluation Gates** | `test_evaluate.py` | 19 | Every model beats its per-cohort baseline; calibration coverage computed; cohorts surfaced not dropped; every model carries an attainable ceiling and every beats-baseline claim an interval. |
| **CRPS** | `test_crps.py` | 5 | The proper-score arithmetic, on synthetic data: a collapsed trio integrates exactly to MAE, the `mcb - dsc + unc` decomposition reconstructs CRPS on imperfectly-calibrated predictions, `unc` matches the unconditional pinball loss at each alpha, and a single quantile level is refused. |
| **Targets** | `test_targets.py` | 3 | Stint-life target is synthesised without leaking `stint_length_laps`; the censoring flag rides in metadata and never becomes a feature; AFT bounds encode censoring as a point vs a half-line. |
| **Survival** | `test_survival.py` | 18 | The AFT contract itself: the +1 shift keeps zero-life stints off log(0), margins round-trip to laps, quantiles bracket the median and widen with scale, censored rows are not punished for over-prediction, and C-index ranks. `aft_params` refuses a non-AFT booster. |
| **Version Contract** | `test_manifest_contract.py` | 18 | The manifest names a version whose artefacts exist, declares the input width the boosters actually take, and matches the copy the browser loads; every `beats_baseline` on the card carries an interval, and that gate is proven to fire. |
| **Attainable Ceilings** | `test_ceiling.py` | 19 | The between-stint variance estimator recovers a known ICC where the naive one inflates it; rolling-window overlap is detected and thinned; the in-sample stint oracle really is the optimum over stint-constant predictors; clustered intervals are wider than lap-grain ones. |
| **Within-stint Attribution** | `test_attribution.py` | 33 | The flatten is a pure within-stint operation; a planted per-lap driver and a planted stint-level one separate under it where a drop cannot tell them apart; the causal summary over laps 1..t provably never reads a later lap; a planted look-ahead result is named unreachable. |
| **Fit Parity** | `test_fit_parity.py` | 24 | The search, the evaluation refit and the production refit fit the same model: the quantile trio's IPW survival weights and the AFT censoring flag reach every fold of every path, sliced to that fold's own rows; a quantile fit offered no weights raises instead of defaulting; and a refit that would replace a version fitted on a different target column is refused. |
| **Search Space** | `test_search_space.py` | 14 | The hyperparameter space is declared once and `_suggest` builds from it, so no suggestion can leave the bounds the pin detector reads; `boundary_params` names a best-params set that stopped on an edge, and is proven to fire at each ceiling, at each floor, and to stay silent in the interior. |
| **Total** | | **189** | |
```mermaid theme={null}
graph TD
subgraph "Leakage Spine (14)"
L1["no leaked columns"]
L2["no forward window"]
L2b["forward-window audit has coverage"]
L2c["self-join horizon detected"]
L3["no AUDIT feature leakage"]
L4["mart contract ⊆ live mart"]
L5["holdout purity"]
L6["MAX+1 holdout"]
L7["target bounded −10/+10"]
L8["no NULL targets (×3 targets)"]
end
subgraph "ONNX Parity (5)"
O1["p10 booster ≈ onnx"]
O2["p50 booster ≈ onnx"]
O3["p90 booster ≈ onnx"]
O4["cliff booster ≈ onnx"]
O5["stint-life booster ≈ onnx"]
end
subgraph "Predict Schema (3)"
P1["output schema 17 cols"]
P2["holdout + envelope flags"]
P3["quantile monotone + probs sum to 1"]
end
subgraph "Evaluation Gates (7)"
E1["all 5 targets evaluated"]
E2["beats baseline (×5 models)"]
E3["cohorts surfaced not dropped"]
end
subgraph "Targets (1)"
T1["stint-life synthesis ≥ 0"]
end
subgraph "Version Contract (10)"
V1["manifest version has a full artefact set"]
V2["manifest ships MODEL_VERSION_DEFAULT"]
V3["declared input width == booster width (×5)"]
V4["input block self-consistent"]
V5["app manifest + onnx sha256 match ml/"]
end
```
## Go deeper
The 14 guards in detail one idea each, with the assertion and why it matters.
The 5 ONNX parity + 3 predict-schema + 1 target tests the mechanical contracts.
The 7 evaluation gates beats-baseline, calibration computed, cohorts surfaced.
The 10 checks that the manifest names the version it shipped, declares the input width the boosters actually take, and matches the copy the browser loads.
The methodology behind the metrics these gates enforce.