The guards
1. No leaked columns (×3 targets)
1. No leaked columns (×3 targets)
Source:
test_features.py::test_no_leaked_columns (parametrized across 3 targets → 3 tests)Assertion: set(X_train.columns) ∩ EXCLUDED_LEAKAGE_COLUMNS == ∅Why it matters: EXCLUDED_LEAKAGE_COLUMNS includes the targets themselves, the driver-skill residual (driver_skill_residual_s), and season identifiers (race_year, driver_id). If any of these leaked into X, the model would be learning the answer. The test parametrizes across three targets (degradation p50, stint-life, cliff classifier) to catch per-target matrix construction bugs.See the Feature Contract for the full excluded-columns list.2. No forward-looking features (SQL audit)
2. No forward-looking features (SQL audit)
Source:
test_features.py::test_no_forward_looking_featuresAssertion: audit_forward_window() == []Why it matters: The mart is built from SQL. A LEAD or FOLLOWING window function in any upstream query would project the next lap’s value into the current row an invisible look-ahead. audit_forward_window walks the compiled dbt manifest with sqlglot and rejects any such clause. An empty return means zero forward-window constructs in the entire mart lineage.The audit runs the full SQL AST, not just a text search it catches LEAD inside CTEs, subqueries, and aliases. It also reads the enclosing scope: a self-join whose ON clause compares a lap index across two aliases of the same table walks a whole forward horizon without a window function anywhere, and is rejected on the same footing.3. The forward-window audit can still see
3. The forward-window audit can still see
Source:
test_features.py::test_forward_window_audit_actually_reads_the_lineageAssertion: every lineage model parses, and all 33 features resolve to a SQL definitionWhy it matters: Coverage, not cleanliness. audit_forward_window returns [] both when it has inspected every feature and found nothing, and when it inspected nothing at all — and for as long as the manifest carried a null compiled_code, it was the second: 0 of 21 models parsed, 0 of 42 features resolved, CLEAN on every run. This test is what makes gate 2 mean something. The failure mode of a gate is silence, and silence is indistinguishable from success.4. Audit features clear the forward window
4. Audit features clear the forward window
Source:
test_features.py::test_audit_features_clear_forward_windowAssertion: No AUDIT_FEATURES (cliff_candidate_flag) appear in any audit_forward_window violationWhy it matters: cliff_candidate_flag is a “label-adjacent” feature it is predictive precisely because it encodes information that co-moves with the cliff. The test asserts that this predictiveness comes from the mart’s temporal conditioning, not from a forward-window leak. If it ever looks ahead in the SQL, it must move to EXCLUDED_LEAKAGE_COLUMNS. (anomaly_class carried the same guard until Phase 9 (2026-09-05) dropped it from the feature contract along with the rest of the context group.)5. No undeclared aggregation scope
5. No undeclared aggregation scope
Source:
test_features.py::test_no_undeclared_aggregation_scopeAssertion: audit_aggregation_scope() == []Why it matters: A forward reach does not need a window function, and it does not need a join predicate either. It can live in the scope of a GROUP BY, where guards 2 and 4 do not look: GROUP BY circuit_slug over every ingested season pools 2024 into what a 2018 row sees, and GROUP BY stint_id hands a lap the median of laps that had not yet run. Neither construct has a LEAD, a FOLLOWING frame or a self-join inequality anywhere in it, so the forward-window audit read both as clean — which it did, for as long as they existed.The rule: every GROUP BY in the mart’s lineage must confine a group to at most one lap (the label’s grain — the mart is one row per valid race lap), or be declared in that model’s schema.yml. A key set pins a lap if it holds lap_id, or a race key beside a lap ordinal, or stint_id beside a lap ordinal. Extra keys only ever shrink a group, so the test is monotone.6. The aggregation audit catches both known shapes
6. The aggregation audit catches both known shapes
Source:
test_features.py::test_aggregation_audit_catches_a_cross_season_pooled_rate, ::test_aggregation_audit_catches_a_bucketed_lap_key, ::test_aggregation_audit_passes_a_lap_pinned_groupAssertion: synthetic SQL reproducing each shape is flagged, with the right severity; a lap-pinned group is notWhy it matters: Falsification, not silence. Two live constructs motivated this audit — a per-circuit rate with no season key, and FLOOR(lap_number / 5.0) * 5.0 AS lap_window grouped as if it were a lap. Both are reproduced as synthetic SQL so the guard is tested against the shapes rather than against the warehouse, and the third test is the negative control: (race_id, lap_number) confines a group to one lap of one race, and a CAST is not a coarsening, so neither may be flagged.7. A bucket cannot be aliased onto a lap key
7. A bucket cannot be aliased onto a lap key
Source:
test_features.py::test_aggregation_audit_rejects_a_bucket_aliased_onto_a_lap_keyAssertion: FLOOR(lap_number / 5.0) * 5.0 AS lap_number, then grouped, is still a violationWhy it matters: The hole a name-based check would leave. If the audit trusted the name of a grouping key, aliasing a five-lap bucket back onto lap_number would buy a pass, and the key set would look like it pins a lap when it spans five — four of them in the future at the first one. The audit resolves each key to what it is computed from, so a derived key is disqualified whatever it is called.8. Declared exemptions are signed, and cannot rot
8. Declared exemptions are signed, and cannot rot
Source:
test_features.py::test_declared_exemptions_are_well_formed, ::test_aggregation_survey_names_the_two_known_instancesAssertion: every declaration carries a known status and a written reason; a known_leak names the item that fixes it; a declaration matching no aggregation in its model is a violationWhy it matters: An exemption is a signature, not a mute button. Declarations live in schema.yml under meta.aggregation_scope_exemptions as data the checker reads — keys, status (accepted or known_leak), reason, and fixed_by when something is still broken — and make ml-features prints the open known_leaks on every run rather than letting them rest quietly in a YAML file. Because a declaration is keyed on the grouping key set, editing the SQL out from under one turns it into a stale exemption and fails the build, instead of silently exempting an aggregation nobody looked at.A second, report-only pass covers the int_*/stg_* models the mart does not read yet. Nothing can leak through a model no feature reads, so failing the build on those would force dozens of declarations that assert nothing — but when one of them is wired into a feature it enters the gate above automatically, and the build stops until someone rules on it.9. Mart contract ⊆ live mart columns
9. Mart contract ⊆ live mart columns
Source:
test_features.py::test_feature_contract_subset_of_martAssertion: Every column in FEATURE_COLUMNS ∪ IDENTIFIER_COLUMNS ∪ mart_target_cols exists in fct_cliff_prediction_featuresWhy it matters: If the transform layer drops or renames a feature column and the ML schema is not updated, the model will silently train on a missing column. This test makes the drift visible immediately the build fails before any training runs. It is the “never again” guard following a previous incident where the mart dropped circuit_key and the ML layer continued loading silently because pandas drops missing columns without warning.10. Holdout purity
10. Holdout purity
Source:
test_features.py::test_holdout_purityAssertion: holdout_season not in training_seasons AND (groups_train < holdout_season).all()Why it matters: If any row from the holdout season appears in any training fold, every metric computed on that season is overstated. This test checks both that the season is not in the training set list and that no training row’s race_year equals or exceeds the holdout season. Both are necessary: the first catches a metadata error; the second catches a data-construction bug.11. No hard-coded holdout year
11. No hard-coded holdout year
Source:
test_features.py::test_no_hardcoded_holdoutAssertion: No numeric literal 2024 or 2025 appears as a code token (not a comment or docstring) in any ml/src/*.py fileWhy it matters: The holdout season is MAX(race_year) + 1. If a developer hard-codes the year, the guard fails silently when the season rolls over 2025 ingests, the literal still says 2025, and the “holdout” is now in-sample. The test tokenizes the source files rather than string-searching, so it distinguishes code from comments correctly.12. Degradation target bounded [−10, 10]
12. Degradation target bounded [−10, 10]
Source:
test_features.py::test_target_boundedAssertion: y_train.between(-10, 10).all() AND (y_train < 0).mean() > 0.2Why it matters: The target next_lap_degradation_jump_detrended_s is bounded by physics: a tyre cannot gain or lose more than 10 seconds per lap in a single step. The upper bound guards against catastrophic outliers that would dominate the loss. The lower bound (> 0.2 fraction negative) checks that the training data still contains enough negative examples if detrending went wrong, the target could become artificially non-negative and the model would never learn tyre recovery.13–15. No NULL targets in training (×3 targets)
13–15. No NULL targets in training (×3 targets)
Source:
test_features.py::test_no_null_targets_in_training (parametrized × 3)Assertion: y_train.notna().all() for each of the three model familiesWhy it matters: The last lap of a stint has no “next lap” its degradation target is NULL. XGBoost raises an error on NaN in y but produces silently wrong results if the NaN is dropped inconsistently. This test asserts that load_features correctly filters NULL-target rows before training for all three families, not just the one being actively developed.The excluded columns
Excluded (22 columns causal leakage / identifiers / targets / training gate):circuit_key, drift_s_per_lap, driver_id, driver_skill_field_s, driver_skill_loro_mean_s, driver_skill_loro_s, driver_skill_proxy_s, driver_skill_residual_proxy_s, driver_skill_residual_s, is_training_eligible, lap_id, laps_until_cliff_class, next_3_lap_cumulative_jump_s, next_5_lap_cumulative_jump_s, next_lap_degradation_jump_detrended_s, next_lap_degradation_jump_s, race_id, race_year, remaining_stint_life_laps, stint_id, stint_length_laps, survival_weight
Relationships
Features & Targets
The forward-window audit in context and why
features.py owns this responsibility.Validation
The adversarial probe that demonstrates the exclusions are non-negotiable
race_year is recoverable at 0.998 accuracy.Feature contract
The full list of 33 features and why each is allowed into
X.CI overview
All 40 tests where the spine sits in the full contract.