> ## Documentation Index > Fetch the complete documentation index at: https://offthepace.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Model validation: leakage spine, CV, and the adversarial probe > Season-grouped TimeSeriesSplit, a five-guard leakage spine, adversarial probes, and the MAX+1-derived holdout policy that flips to a true reveal when 2025 ingests. Forward-looking targets make leakage the defining risk of the machine layer. A model that accidentally sees the future or the answer will look good on paper and fail the moment it runs on live data. Off The Pace defends against this with a **leakage spine**: a set of guards baked into tests and CI that prove the model never sees what it should not. ## Season-grouped time-series validation Tyre data is a time series. A random train/test split would let the model train on 2024 laps and then "predict" 2022 which is not prediction at all. Instead, validation uses a **season-grouped `TimeSeriesSplit`** with an expanding window across five splits: whole seasons move together, and each fold validates on the season(s) immediately after those it trained on. ```mermaid theme={null} gantt title Season-grouped expanding-window cross-validation dateFormat YYYY axisFormat %Y section Fold 1 Train 2018-2019 :2018, 2020 Validate 2020 :crit, 2020, 2021 section Fold 2 Train 2018-2020 :2018, 2021 Validate 2021 :crit, 2021, 2022 section Fold 3 Train 2018-2021 :2018, 2022 Validate 2022 :crit, 2022, 2023 section Fold 4 Train 2018-2022 :2018, 2023 Validate 2023 :crit, 2023, 2024 section Fold 5 · headline Train 2018-2023 :2018, 2024 Validate 2024 :crit, 2024, 2025 ``` The final fold trains on 2018–2023 and validates on **2024** the most recent, holdout-shaped unseen season. That final-fold score is the evaluation headline reported in the [model metrics table](/ml/models#headline-results). **2025 is the planned true out-of-sample holdout.** The holdout season is data-derived, not hard-coded: `HOLDOUT_SEASON = MAX(race_year) + 1`. Today that resolves to 2025, which has no rows yet. The moment 2025 data ingests, the evaluation logic detects the populated holdout and switches to a true out-of-sample reveal **with zero code change**. The split year is never a hard-coded literal the test suite enforces this. *** ## The leakage spine five guards | Guard | What it proves | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | **No leaked columns** | Targets, driver-skill signals, and season identifiers never enter the feature matrix | | **No forward-looking features** | A `sqlglot` audit walks the compiled mart and all ancestors, rejecting any `LEAD` or `FOLLOWING` window function | | **Holdout purity** | No holdout-season row appears in any training fold | | **No hard-coded holdout** | The split is `MAX+1`-derived, never a literal year | | **Bounded / non-null targets** | Degradation ∈ \[−10, 10]; no NULL-target rows enter training | These five guards are part of the **28-test suite** and run automatically in CI on every change. Reproduce locally with `make ml-test`. Each guard is explained in depth on the [Leakage Spine page](/ml/ci/leakage-spine). *** ## The adversarial leakage probe The two most consequential exclusions `driver_id` and `race_year` are justified by **demonstration, not assertion**. A throwaway XGBoost model is trained to recover `race_year` from the remaining features. It succeeds at **0.998 accuracy** against a majority-class baseline of 0.168 a lift of 0.83. That is the point. Constructor identities and compound generations carry such a strong residual temporal signal that `race_year` is almost perfectly recoverable from them. Including `race_year` as a feature would create a backdoor to the season, and through it to outcomes the model should not know. The same logic applies to `driver_id`, which would let the trees relearn per-driver skill the very signal the decomposition works to isolate into `driver_skill_residual_s`. Both are correctly excluded. A feature that *predicts the season* is a feature that predicts the answer. Temporal leakage is silent it never throws an error, it just inflates every offline number and collapses the moment the model meets a season it has not memorised. The probe is the tripwire that makes the exclusion non-negotiable. *** ## Metric definitions Each model family is judged on a single headline metric, computed directly from the implementations in `ml/src/train.py` and `ml/src/evaluate.py`. The quantile trio is scored by the same **pinball loss** it is trained on, at $\alpha \in \{0.1, 0.5, 0.9\}$. For a residual $u = y - \hat{y}$: $\ell_\alpha(y,\hat{y}) = \max\!\big(\alpha\,u,\;(\alpha-1)\,u\big)$ Lower is better. Because the loss is the training objective, beating the empirical-percentile baseline on it is direct evidence the model has learned conditional structure the cohort average has not. Full derivation in [Models → Objective functions](/ml/models#objective-functions). The stint-life regressor is scored by **root-mean-square error** in laps: $\mathrm{RMSE} = \sqrt{\tfrac{1}{n}\sum_i (y_i - \hat{y}_i)^2}$ Lower is better. RMSE punishes large misses quadratically, which is the right risk profile for a planning horizon: being ten laps wrong is far worse than being one lap wrong ten times. The cliff classifier is scored by **macro-averaged F1** the *unweighted* mean of the four per-class F1 scores: $\text{F1}_c = \frac{2\,P_c\,R_c}{P_c + R_c}, \qquad \text{macro-F1} = \frac{1}{C}\sum_{c=1}^{C}\text{F1}_c$ where $P_c$ and $R_c$ are precision and recall for class $c$ and $C = 4$. Averaging *unweighted* (rather than by class support) is deliberate: it gives the rare `0_to_2` window the same say as the dominant `6_plus` class, so a model that quietly ignored imminent cliffs would be punished hard. This is also why macro-F1 sets a far lower ceiling than raw accuracy would. *** ## Calibration The degradation quantile trio claims an 80% prediction interval from `[p10, p90]`. The raw quantiles are **already calibrated** empirical coverage lands within \~0.010 of nominal on the 2024 evaluation fold. A split-conformal (CQR) correction is computed post-hoc and confirms calibration with a finite-sample guarantee. In plain terms: the model is right approximately four times in five, and the interval averages \~1.24 seconds wide. The full LaTeX derivation of the conformity score, the CQR quantile formula, and the coverage table live on the [Calibration page](/ml/calibration). *** ## Cohorts surfaced, never dropped Beyond the global headline, every model is scored across cohorts: compound, circuit, constructor, rain-lap, and season. Underperforming cells are recorded to the model card rather than hidden. Most are the near-oracle stint-life baseline winning on specific circuits (the Dutch, Japanese, and Las Vegas Grands Prix among them). The contract is explicit: surface the losses, do not hide them. The full cohort table and the context for each loss live on the [Cohorts page](/ml/cohorts). *** *** ## Known limitations **No live 2025 holdout yet.** Evaluation headlines are the final time-series CV fold (2024). They become a true out-of-sample reveal automatically when 2025 data ingests but until that moment, they are cross-validation, not a held-out season. * **The cliff classifier is the weakest model but not against the right denominator.** Macro-F1 ≈ 0.40 decisively beats the majority prior, and read against 1.0 that looks modest. 1.0 is not reachable here: against an oracle handed the stint id and nothing else, the model scores **114% of the ceiling**. What limits it is the label, not the learner `laps_until_cliff_class` is a first-crossing scan over an unbounded cliff polynomial. The 100 most-confident misses are exported for study. * **Every headline on this page is now reported against what is attainable, not against 1.0.** A pinball loss of 0.199 and a macro-F1 of 0.404 are uninterpretable as fractions of a perfect score. Each model card entry carries the between-stint variance share of its target and the fraction of the reachable improvement the model captures. Three of the five score *past* their stint-level ceiling, which is a statement about the ceiling: it only ever bounded predictors that are constant within a stint. * **Every `beats_baseline` claim carries an interval.** The effective sample is \~7,100 stints, not \~137,000 laps, so a lap-grain point estimate overstates certainty. Each margin is reported as a paired t across the five season folds and as a bootstrap that resamples whole stints; a claim that does not clear its own interval is listed rather than removed. * **About half the degradation model's within-stint signal is not tyre state.** The models reach past what stint identity alone can supply, so it matters what they reach with. Measured by replacing each feature group with its per-stint average which keeps the group's stint-level information and removes only its lap-to-lap variation roughly half of that signal (50.7%) is the in-stint age ramp, and roughly half is driver push (29.6%) plus traffic (19.7%). Only contributions clearing a measured seed-refit noise floor are counted. The `dirty_air` group's contribution is **95.2% lap-level**: the model is reading which *laps* ran behind a car, not which stints did. Reproduce with `make ml-attribution`. This measures what the models *use*, which is not the same as what a feature could *supply*. The per-stint average is taken over the whole stint, so on a forward-looking target it includes laps that had not run when the row is scored fine for asking what the model reads, inadmissible in a feature. Every share quoted above is a case where the lap-to-lap variation won *against* that advantage, so they stand as lower bounds; results pointing the other way are re-measured over laps 1..t before they are believed, and the run reports that causal arm beside every group that clears the noise floor. * **v1 hyperparameters come from a reduced tuning budget.** Running `make ml-tune` (50 trials / 5 folds / full data) only improves them no code change required and should precede any production blessing. * **\~47% of laps have a null cliff-onset prior** (2018 legacy compounds, un-fit circuits). XGBoost handles the missingness natively via default-direction splits rather than imputation this is documented and its ONNX parity is proven but those laps lean less on the strongest prior feature.