Evaluation gates: beats-baseline, calibration, and cohorts

The tests in test_evaluate.py that assert every model beats its per-cohort baseline, cohorts are surfaced not dropped, every model carries an attainable ceiling, and every win carries an interval.

The 19 evaluation tests in test_evaluate.py (seven test functions, four of them parametrized across the five models) enforce the properties that make the model card trustworthy: all five models beat honest baselines, every headline has a denominator and an interval, and no cohort loss is silently hidden. These tests run against ml/artefacts/evaluation_metrics.json, produced by make ml-evaluate.

These tests are skipped when evaluation_metrics.json is absent (i.e. in CI pull requests that do not retrain). They are always run before a release. To run them locally: make ml-evaluate && make ml-test.

​

The 19 tests

Gate 1: All five targets evaluated

Source: test_evaluate.py::test_all_targets_evaluated

Assertion: set(report["models"]) == {target.name for target in PRODUCTION_TARGETS}

Why it matters: if evaluate.py silently skips a model (e.g. a name mismatch after a rename), the missing model’s metrics would not appear in the card, the model could ship without a baseline comparison, and the CI would pass. This gate ensures the evaluation ran for all five not just some of them.

Gates 2–6: Every model beats its baseline (×5 models)

Source: test_evaluate.py::test_model_beats_baseline_overall (parametrized × 5)

Assertion (for each model):

• beats_baseline flag in the report matches the raw metric comparison
• model_score is better than baseline_score in the correct direction (↓ for pinball/RMSE, ↑ for macro-F1)

Why it matters: the beats_baseline flag in the card is set by evaluate.py. This test re-derives the comparison from the raw numbers to catch any inversion in the direction logic (e.g. accidentally using < instead of > for macro-F1). If the flag and the raw numbers disagree, one of them is wrong and the gate will not accept either.

The baselines are:

• Quantile trio: the empirical p10/p50/p90 of the degradation target within the same (compound, circuit, lap-age-bucket) cell
• Cliff classifier: the majority-class precision within the same (compound, circuit) cell
• Stint-life regressor: the mean remaining_stint_life_laps within the same (compound, circuit, lap-in-stint-bucket) cell

These are genuinely non-trivial baselines they know the typical behaviour for this compound and this circuit. Beating them means the model adds real conditional signal.

See Models for the current scores.

Gate 7: Cohorts surfaced, not dropped

Source: test_evaluate.py::test_underperforming_cohorts_present

Assertion: every model’s entry in the report contains an underperforming_cohorts key

Why it matters: a model that silently skips the cohort-level evaluation (e.g. because an exception was swallowed) would pass every global metric gate while hiding per-cohort failures. This gate checks that the evaluation ran the cohort loop for every model and wrote the result even if that result is an empty list (which would mean the model beats the baseline in every cell, a suspicious but plausible outcome).

See Cohorts for the current underperforming cells and their context.

Gates 8–12: Every model reports an attainable ceiling (×5 models)

Source: test_evaluate.py::test_every_model_reports_an_attainable_ceiling (parametrized × 5)

Assertion (for each model):

• the attainable block exists and carries no error
• variance.between_stint_share lies in [0, 1]
• the corrected (ANOVA) between-stint share does not exceed the naive share, unless the target’s within-stint variation is deterministic, as it is for stint life

Why it matters: a headline score with no denominator cannot say whether to keep tuning or stop. This gate requires every model’s report to say how much of the variance a stint-aware predictor could reach. The ordering check catches an estimator that has regressed to the naive one, which overstates the ceiling.

Gates 13–17: Every win carries an interval (×5 models)

Source: test_evaluate.py::test_every_beats_baseline_claim_has_an_interval (parametrized × 5)

Assertion (for each model that claims beats_baseline):

• the interval block exists and carries no error
• the fold-paired result has at least two folds, a non-null p_value, and ci_low < ci_high
• a beats_baseline_significant verdict is recorded

Why it matters: the effective sample is the number of stints, not the number of laps, so a lap-grain point estimate can look decisive when it is not. A model that does not claim a win is skipped. A model that does must publish the interval and the verdict that go with it.

Gate 18: Stint grain recorded, folds stint-pure

Source: test_evaluate.py::test_stint_grain_is_recorded_and_the_folds_are_stint_pure

Assertion: the report has a stint_grain block, stints_straddling_a_season_fold == 0, and n_stints < n_rows

Why it matters: laps within a stint are not independent, so the report must record how many stints stand behind the lap count. The zero-straddle check confirms the season-grouped folds never split a stint, which is what shows the cross-validation split was not the source of any clustering problem. A straddling stint would mean a stint belongs to two seasons, which should be impossible.

Gate 19: Claims inside noise is present, even when empty

Source: test_evaluate.py::test_claims_inside_noise_is_present_even_when_empty

Assertion: the report has a claims_inside_noise key, and its entries are a subset of the evaluated model names

Why it matters: a claim that fails its own interval is the most informative line in the report. An absent key and “every claim survived” must not look the same, so the key has to exist even when the list is empty.

​

The relationship to the model card

The evaluation gates check evaluation_metrics.json; the model card (model_card.yml) is written by ml-card from the same source. The reference docs (docs/reference/ml/degradation-model.mdx) are generated from the card. The chain is:

make ml-evaluate  →  evaluation_metrics.json  →  ml-card  →  model_card.yml  →  ml-reference  →  docs
      ↑
 test_evaluate.py gates this output

Any model that fails an evaluation gate cannot produce a valid card, and therefore cannot produce valid reference docs. The gates enforce the property “published numbers are honest” at the point where the metrics are first computed.

​

Relationships

Validation

The CV scheme, baselines, and calibration methodology these gates enforce.

Cohorts

The surfaced underperforming cohort cells and what they mean.

Models

The headline numbers the beats-baseline gates are checking.

CI overview

All 270 tests where these 19 sit in the full contract.