Skip to main content
Temporal leakage is silent. It never throws an error. It inflates every offline metric and collapses the moment the model meets a season it has not memorised. The leakage spine is a set of 34 CI tests (26 test functions, three of them parametrized across targets) that make leakage structurally impossible not just “unlikely” or “checked once.”
A model that accidentally sees the future looks better on paper than a model that does not. There is no natural corrective: the offline metrics reward the leak, the model ships, and the live numbers disagree. The only reliable defence is to make the leak structurally impossible and then test that impossibility.

​
The guards

Source: test_features.py::test_no_leaked_columns (parametrized across 3 targets → 3 tests)Assertion: set(X_train.columns) ∩ EXCLUDED_LEAKAGE_COLUMNS == ∅Why it matters: EXCLUDED_LEAKAGE_COLUMNS includes the targets themselves, the driver-skill residual (driver_skill_residual_s), and season identifiers (race_year, driver_id). If any of these leaked into X, the model would be learning the answer. The test parametrizes across three targets (degradation p50, stint-life, cliff classifier) to catch per-target matrix construction bugs.See the Feature Contract for the full excluded-columns list.
Source: test_features.py::test_no_forward_looking_featuresAssertion: audit_forward_window() == []Why it matters: The mart is built from SQL. A LEAD or FOLLOWING window function in any upstream query would project the next lap’s value into the current row an invisible look-ahead. audit_forward_window walks the compiled dbt manifest with sqlglot and rejects any such clause. An empty return means zero forward-window constructs in the entire mart lineage.The audit runs the full SQL AST, not just a text search it catches LEAD inside CTEs, subqueries, and aliases. It also reads the enclosing scope: a self-join whose ON clause compares a lap index across two aliases of the same table walks a whole forward horizon without a window function anywhere, and is rejected on the same footing.
Source: test_features.py::test_forward_window_audit_actually_reads_the_lineageAssertion: every lineage model parses, and all 39 features resolve to a SQL definitionWhy it matters: Coverage, not cleanliness. audit_forward_window returns [] both when it has inspected every feature and found nothing, and when it inspected nothing at all — and for as long as the manifest carried a null compiled_code, it was the second: 0 of 21 models parsed, 0 of 42 features resolved, CLEAN on every run. This test is what makes gate 2 mean something. The failure mode of a gate is silence, and silence is indistinguishable from success.
Source: test_features.py::test_audit_features_clear_forward_windowAssertion: No AUDIT_FEATURES (cliff_candidate_flag) appear in any audit_forward_window violationWhy it matters: cliff_candidate_flag is a “label-adjacent” feature it is predictive precisely because it encodes information that co-moves with the cliff. The test asserts that this predictiveness comes from the mart’s temporal conditioning, not from a forward-window leak. If it ever looks ahead in the SQL, it must move to EXCLUDED_LEAKAGE_COLUMNS. (anomaly_class carried the same guard until it was dropped from the feature contract along with the rest of the context group.)
Source: test_features.py::test_no_undeclared_aggregation_scopeAssertion: audit_aggregation_scope() == []Why it matters: A forward reach does not need a window function, and it does not need a join predicate either. It can live in the scope of a GROUP BY, where guards 2 and 4 do not look: GROUP BY circuit_slug over every ingested season pools 2024 into what a 2018 row sees, and GROUP BY stint_id hands a lap the median of laps that had not yet run. Neither construct has a LEAD, a FOLLOWING frame or a self-join inequality anywhere in it, so the forward-window audit read both as clean — which it did, for as long as they existed.The rule: every GROUP BY in the mart’s lineage must confine a group to at most one lap (the label’s grain — the mart is one row per valid race lap), or be declared in that model’s schema.yml. A key set pins a lap if it holds lap_id, or a race key beside a lap ordinal, or stint_id beside a lap ordinal. Extra keys only ever shrink a group, so the test is monotone.
Source: test_features.py::test_aggregation_audit_catches_a_cross_season_pooled_rate, ::test_aggregation_audit_catches_a_bucketed_lap_key, ::test_aggregation_audit_passes_a_lap_pinned_groupAssertion: synthetic SQL reproducing each shape is flagged, with the right severity; a lap-pinned group is notWhy it matters: Falsification, not silence. Two live constructs motivated this audit — a per-circuit rate with no season key, and FLOOR(lap_number / 5.0) * 5.0 AS lap_window grouped as if it were a lap. Both are reproduced as synthetic SQL so the guard is tested against the shapes rather than against the warehouse, and the third test is the negative control: (race_id, lap_number) confines a group to one lap of one race, and a CAST is not a coarsening, so neither may be flagged.
Source: test_features.py::test_aggregation_audit_rejects_a_bucket_aliased_onto_a_lap_keyAssertion: FLOOR(lap_number / 5.0) * 5.0 AS lap_number, then grouped, is still a violationWhy it matters: The hole a name-based check would leave. If the audit trusted the name of a grouping key, aliasing a five-lap bucket back onto lap_number would buy a pass, and the key set would look like it pins a lap when it spans five — four of them in the future at the first one. The audit resolves each key to what it is computed from, so a derived key is disqualified whatever it is called.
Source: test_features.py::test_declared_exemptions_are_well_formed, ::test_the_sc_hazard_instance_left_the_survey_by_being_fixed_not_by_being_hidden, ::test_the_corner_instance_left_the_survey_by_being_fixed_not_by_being_hiddenAssertion: every declaration carries a known status and a written reason; a known_leak names the item that fixes it; a declaration matching no aggregation in its model is a violationWhy it matters: An exemption is a signature, not a mute button. Declarations live in schema.yml under meta.aggregation_scope_exemptions as data the checker reads — keys, status (accepted or known_leak), reason, and fixed_by when something is still broken — and make ml-features prints the open known_leaks on every run rather than letting them rest quietly in a YAML file. Because a declaration is keyed on the grouping key set, editing the SQL out from under one turns it into a stale exemption and fails the build, instead of silently exempting an aggregation nobody looked at.A second, report-only pass covers the int_*/stg_* models the mart does not read yet. Nothing can leak through a model no feature reads, so failing the build on those would force dozens of declarations that assert nothing — but when one of them is wired into a feature it enters the gate above automatically, and the build stops until someone rules on it.
Source: test_features.py::test_feature_contract_subset_of_martAssertion: Every column in FEATURE_COLUMNS ∪ IDENTIFIER_COLUMNS ∪ mart_target_cols exists in fct_cliff_prediction_featuresWhy it matters: If the transform layer drops or renames a feature column and the ML schema is not updated, the model will silently train on a missing column. This test makes the drift visible immediately the build fails before any training runs. It is the “never again” guard following a previous incident where the mart dropped circuit_key and the ML layer continued loading silently because pandas drops missing columns without warning.
Source: test_features.py::test_holdout_purityAssertion: holdout_season == holdout_config.HOLDOUT_SEASON, holdout_season not in training_seasons AND (groups_train < holdout_season).all(); when holdout rows exist they all belong to that one seasonWhy it matters: If any row from the holdout season appears in any training fold, every metric computed on that season is overstated. This test checks that the season is the pinned one, that it is not in the training set list, and that no training row’s race_year equals or exceeds it. Both are necessary: the first catches a metadata error; the second catches a data-construction bug.
Source: test_features.py::test_no_hardcoded_holdoutAssertion: No numeric literal 2024 or 2025 appears as a code token (not a comment or docstring) in any ml/src/*.py fileWhy it matters: The holdout season is pinned in one place, holdout_config.HOLDOUT_SEASON, and every other module reads it from there. If a developer hard-codes a year elsewhere, the guard fails silently when the season rolls over: the literal goes stale and the “holdout” is now in-sample. The pinned file is the only exemption. The test tokenizes the source files rather than string-searching, so it distinguishes code from comments correctly.
Source: test_features.py::test_target_boundedAssertion: y_train.between(-TARGET_BOUND, TARGET_BOUND).all() AND (y_train < 0).mean() > 0.2, and remaining_stint_life_laps >= 0Why it matters: The degradation target is bounded at source by the clip the mart applies (±50 s for the current five-lap cumulative column). The bound guards against catastrophic outliers that would dominate the loss. The lower bound (> 0.2 fraction negative) checks that the training data still contains enough negative examples if detrending went wrong, the target could become artificially non-negative and the model would never learn tyre recovery.
Source: test_features.py::test_no_null_targets_in_training (parametrized × 3)Assertion: y_train.notna().all() for each of the three model familiesWhy it matters: The last lap of a stint has no “next lap” its degradation target is NULL. XGBoost raises an error on NaN in y but produces silently wrong results if the NaN is dropped inconsistently. This test asserts that load_features correctly filters NULL-target rows before training for all three families, not just the one being actively developed.
Source: test_features.py::test_forward_window_audit_catches_a_self_join_horizonAssertion: with laps_until_cliff as the only feature, audit_forward_window() returns a violation containing “self-join inequality”Why it matters: The cliff scan walks a whole horizon through a.lap_in_stint < f.lap_in_stint in a JOIN ... ON, with no LEAD and no FOLLOWING frame anywhere. A feature built that way passed the audit silently until the walker learned to read the enclosing scope. This test shows the audit still flags the shape it was extended for.
Source: test_features.py::test_the_sc_hazard_instance_left_the_survey_by_being_fixed_not_by_being_hiddenAssertion: int_sc_hazard_history no longer appears in survey_aggregation_scope(), the survey no longer reports its all-time “pools every ingested season” shape, the model is in the mart lineage, and audit_aggregation_scope() has no violation naming itWhy it matters: A model can drop off the report-only survey for two reasons: its defect was removed, or a feature started reading it. The second alone would hide a model that still leaks. The test requires both, so the model is now keyed by circuit and season with every aggregate ending one season back, it sits inside the audited lineage, and the audit is clean on it. The survey check is scoped to this model on purpose, because other models outside the lineage still report the pooled shape and none of them feeds a feature.
Source: test_features.py::test_the_sc_hazard_join_key_still_carries_the_seasonAssertion: the SQL of fct_cliff_prediction_features joins int_sc_hazard_history on sch.circuit_slug and on r.race_year = sch.seasonWhy it matters: The hazard is point-in-time as of the start of a season, so the season is half the key. A join on circuit alone would hand a 2018 row a hazard estimated partly from 2024 and fan the mart out five to seven times, and neither the window walker nor the aggregation walker reads join keys. The test checks the SQL text structurally, so it fails on the edit that causes the bug rather than on the rebuild that reveals it.
Source: test_features.py::test_the_corner_instance_left_the_survey_by_being_fixed_not_by_being_hiddenAssertion: int_corner_skill_residuals is absent from the survey, both it and int_lap_corner_inputs are in the mart lineage, and audit_aggregation_scope() has no violation naming eitherWhy it matters: The same two-reason check as the safety-car hazard. The five-lap block bucket that once made this model’s groups reach forward was replaced by a trailing range window, and the model entered the mart through int_lap_corner_inputs. Asserting both halves stops a model from leaving the survey by being wired into a feature while still carrying the defect.
Source: test_features.py::test_baseline_observations_n_is_never_a_featureAssertion: "baseline_observations_n" not in FEATURE_COLUMNSWhy it matters: The column ships in the mart because the thermal features’ NULLs are deterministic on it, so consumers need it to condition on that missingness. A feature add-ablation measured it and no model family cleared its reseed floor. Its rank correlation with lap_in_stint is 0.97, so it is easy to add back by mistake. A future change that wants it in X has to re-run the ablation and delete this test on purpose.
Source: test_features.py::test_target_bound_follows_the_target_columnAssertion: TARGET_BOUND == TARGET_BOUND_BY_COLUMN[DEGRADATION_TARGET], and every quantile model’s source column has an entry in TARGET_BOUND_BY_COLUMNWhy it matters: The bound is a fact about the clip on whichever column is modelled, not a constant that fits the one-lap target. Carrying a ±10 bound over to the five-lap column, which is clipped at ±50, would put a range into the manifest that is five times too tight and that nothing downstream checks.
Source: test_features.py::test_target_bound_is_the_clip_the_warehouse_appliesAssertion: the largest absolute value in the degradation training target is more than 90% of TARGET_BOUNDWhy it matters: A bound that is five times too loose passes the bounded-target test forever. This one requires the data to actually reach the bound, which shows the bound is the SQL clip and not an arbitrary envelope drawn around it.
Source: test_features.py::test_target_columns_are_never_features (parametrized × 3)Assertion: the model’s source column is not in FEATURE_COLUMNS; for every model except stint life it is in EXCLUDED_LEAKAGE_COLUMNS; and no column in TARGET_HORIZON_LAPS (every alternative-horizon degradation column) is in FEATURE_COLUMNSWhy it matters: When the modelled column moves between horizons, the column just vacated can quietly become available as a predictor. The test bars the target of each production model and keeps every alternative-horizon degradation column barred whether or not it is the one being modelled.
Source: test_features.py::test_check_cli_defaults_persist_encoders_falseAssertion: running features.py --check calls _check with persist_encoders=FalseWhy it matters: --check used to hard-code persist_encoders=True, so a local check against the dev warehouse overwrote the shipped encoders.json with whatever vocabulary the holdout resolver returned, which may not match the one baked into the shipped ONNX models. The test replaces _check with a stub, so it never touches the real file.
Source: test_features.py::test_check_cli_persist_encoders_flag_threads_throughAssertion: features.py --check --persist-encoders calls _check with persist_encoders=TrueWhy it matters: This is the other direction of the previous guard. It shows the escape hatch reaches _check and is not merely declared and ignored.
Source: test_features.py::test_load_features_does_not_persist_encoders_by_defaultAssertion: with ENCODERS_PATH redirected to a scratch file, load_features(...) writes nothing, and load_features(..., persist_encoders=True) writes the fileWhy it matters: This is the write path that both --check and training go through. Testing both directions shows the guard in load_features itself is correct, again without touching the shipped ml/models/encoders.json.

​
The excluded columns

Read-only mart contract. ML never writes the warehouse or the app. It reads fct_cliff_prediction_features read-only. The columns below are excluded from every model’s feature matrix (EXCLUDED_LEAKAGE_COLUMNS in ml/src/schema.py), asserted by the leakage-spine tests every build.
Excluded (22 columns causal leakage / identifiers / targets / training gate): circuit_key, drift_s_per_lap, driver_id, driver_skill_field_s, driver_skill_loro_mean_s, driver_skill_loro_s, driver_skill_proxy_s, driver_skill_residual_proxy_s, driver_skill_residual_s, is_training_eligible, lap_id, laps_until_cliff_class, next_3_lap_cumulative_jump_s, next_5_lap_cumulative_jump_s, next_lap_degradation_jump_detrended_s, next_lap_degradation_jump_s, race_id, race_year, remaining_stint_life_laps, stint_id, stint_length_laps, survival_weight

​
Relationships

Features & Targets

The forward-window audit in context and why features.py owns this responsibility.

Validation

The adversarial probe that demonstrates the exclusions are non-negotiable race_year is recoverable at 0.998 accuracy.

Feature contract

The full list of 39 features and why each is allowed into X.

CI overview

All 270 tests where the spine sits in the full contract.