Conceptual foundation: Methodology · feature source: fct_cliff_prediction_features.
Intended use
Race-strategy analysis and the Off the Pace web app (application layer): surfacing when a stint is about to fall off the degradation cliff and how much pace a driver is leaving on track. Not a betting or safety system.Models & headline performance
Every model is evaluated against a strong per-cohort baseline; all five beat it. Validation: season-grouped TimeSeriesSplit (expanding window, n_splits=5); whole seasons move together, the final fold validates on 2024.
Eval headline on the cv_final_fold (season 2025); CV is the season-grouped TimeSeriesSplit mean.
Of attainable is the column to read. A pinball of 0.20 or a macro-F1 score of 0.40 says nothing on its own, because neither is a ratio against anything reachable — 1.0 is not a target any model of this data could hit. The column divides each headline’s improvement over an uninformed floor by the improvement a predictor with perfect stint-level knowledge could achieve.
A ⚑ marks a model that scores past that ceiling. It is not an error and not a broken metric: laps inside a stint share a compound, a car, a circuit, a fuel load and a driver, so a stint-level ceiling bounds only predictors that are constant within a stint. Clearing it is proof that the model is using within-stint variation — and that no ceiling has yet been established for it.
Per-model detail
degradation_regressor_p10
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.1) - Train rows: 83,424 · fit: 8.32s
- Artefacts:
ml/models/degradation_regressor_p10_v14.bst+ parity-testedml/models/degradation_regressor_p10_v14.onnx
degradation_regressor_p50
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.5) - Train rows: 83,424 · fit: 13.85s
- Artefacts:
ml/models/degradation_regressor_p50_v14.bst+ parity-testedml/models/degradation_regressor_p50_v14.onnx
degradation_regressor_p90
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.9) - Train rows: 83,424 · fit: 5.25s
- Artefacts:
ml/models/degradation_regressor_p90_v14.bst+ parity-testedml/models/degradation_regressor_p90_v14.onnx
cliff_classifier
- Family / kind: cliff_classifier / classification
- Objective:
multi:softprob - Train rows: 105,525 · fit: 63.41s
- Artefacts:
ml/models/cliff_classifier_v14.bst+ parity-testedml/models/cliff_classifier_v14.onnx
stint_life_regressor
- Family / kind: stint_life_regressor / survival
- Objective:
survival:aft - Train rows: 124,501 · fit: 3.18s
- Artefacts:
ml/models/stint_life_regressor_v14.bst+ parity-testedml/models/stint_life_regressor_v14.onnx
Features
39 features in 7 ablation groups. Categoricals are ordinal-encoded from the training map (NULL/unseen → −1).Leakage guards
driver_id and race_year are deliberately excluded: an adversarial probe recovers race_year from the remaining features at 0.947 accuracy (majority baseline 0.148)-constructor identity and compound generation encode the season, so race_year would be a backdoor. driver_id would let the trees relearn per-driver skill, the exact signal the mart strips via driver_skill_residual_s. Excluded (identity / target / leakage):circuit_key, drift_s_per_lap, driver_id, driver_skill_field_s, driver_skill_loro_mean_s, driver_skill_loro_s, driver_skill_proxy_s, driver_skill_residual_proxy_s, driver_skill_residual_s, is_training_eligible, lap_id, laps_until_cliff_class, next_3_lap_cumulative_jump_s, next_5_lap_cumulative_jump_s, next_lap_degradation_jump_detrended_s, next_lap_degradation_jump_s, race_id, race_year, remaining_stint_life_laps, stint_id, stint_length_laps, survival_weight
Validation
Baselines
Attainable ceilings
Skill is reported as a fraction of the ATTAINABLE quantity, not of 1.0. For the quantile trio the denominator is analytic — under a Gaussian shape a predictor that knew the stint could remove at most 1 - sqrt(1 - ICC) of the expected pinball, where ICC is the between-stint variance share estimated by one-way ANOVA. For the classifier it is a per-stint majority-class oracle; for stint life it is the censored AFT likelihood at a perfect prediction, which is a hard bound. A fraction above 1.0 means the model reaches past stint identity into within-stint variation.
Between-stint share is a one-way random-effects (ANOVA) estimate. The naive column is
var(per-stint means) / var(column), which counts within-stint scatter as between-stint signal at ~19 laps per stint; it is shown so the correction stays visible rather than silently applied.
Intervals on every claim
Every beats_baseline claim carries a paired-t interval over the season-grouped CV folds (n=5) plus a percentile bootstrap that resamples whole STINTS on the evaluation fold. Laps inside a stint share a compound, a car, a circuit, a fuel load and a driver, so a lap-grain interval understates the standard error; the ratio between the two is published per model rather than assumed.
Every
beats_baseline claim clears its own interval.
Effective sample: 6,368 stints over 83,424 laps (13.1 laps per stint). 0 stints straddle a season fold boundary, so the CV split was never the problem — every interval computed at lap grain was.
Calibration (quantile interval)
Nominal coverage 0.8 → raw [p10,p90] empirical 0.8216, split-conformal 0.8154 (mean interval width 5.3568s).Adversarial leakage probe
A throwaway model recoversrace_year from the features at 0.9474 vs a majority baseline of 0.1478-X carries strong residual temporal signal-race_year recoverable far above chance; race_year is correctly excluded from features.
Feature importance (SHAP vs permutation)
Underperforming cohorts
Surfaced, never dropped. Specific circuit/constructor cells where the model trails its non-leakage cohort-mean baseline; the model wins comfortably overall. 7 cohort cells lose to baseline (surfaced, never dropped):Reproducibility
- Seed:
20260528· dataset fingerprint:5dbd8e51502a5f83… - Libraries: xgboost 3.4.1, sklearn 1.9.1
- ONNX parity: all 5 boosters round-trip to ONNX within atol=1e-5 on a NaN-bearing sample (M2/M4)
- Reproduce:
make ml-all
Limitations
- LABEL SPINE BUMPED AFTER THESE ARTEFACTS WERE TRAINED (WI-01 + WI-12, 2026-09-27): NOT COMPARABLE AT FIXED TARGET. The residual that both labels difference (
driver_skill_residual_s, behindnext_5_lap_cumulative_jump_sandlaps_until_cliff_class) now has five additive terms, not seven: pace_delta = fuel + compound + constructor + dirty-air tax + driver skill. The field base is fuel- AND compound-neutral, and rubber and ambient track state live only inside the base instead of being subtracted a second time. In the same bump: laps with no measured base or an unknown tyre cost carry a NULL label instead of a fabricated one;theta_airis 0.331 s/lap, the within-stint estimate (stint fixed effects plus tyre-age bins) fitted with tyre cost subtracted on a declared season window (2018-2025) and frozen as a declared value; the circuit x constructor interaction is out of the constructor term;compound_grip_peak(a unitless ratio) is out ofexpected_compound_pace_s; SC/VSC/red-flag/restart laps and the lap after an SC or red-flag restart are out ofis_training_eligible. WI-01 and WI-12 are one bump:theta_airpassed through an interim pooled 0.503 s/lap before the W40 ruling set 0.331, and the final label carries 0.331. Every number on this card was measured on the PRE-BUMP label. No post-bump number may be compared with it: a headline on the new label is compared only against this configuration rebuilt on the final (0.331) label. Evidence:_roadmap/_fixes/wi/WI-01-label-spine.md,_roadmap/_fixes/wi/WI-12-06b-remeasure.md. - STINT-LIFE HYPERPARAMETERS ARE WORK ITEM 10e’s S1x, AND THEY WERE LANDED ON A DECISION (D4), NOT ON A CLEARED GATE. What changed: n_estimators 200 -> 100, learning_rate 0.02631 -> 0.02192, aft_loss_distribution_scale 0.80 -> 0.754, plus reg_alpha/min_child_weight/subsample/colsample_bytree. max_depth stayed at 8 - this is a SHRINKAGE change, not the capacity change the earlier diagnosis (10d) predicted. Why: the superseded params were selected on the MIXTURE AFT NLL with the 2024 evaluation season inside the validation folds, which is three defects in the search rather than a property of the model. WHAT IS ESTABLISHED AT 95%: on the green-flag-pit stratum - 9,149 eval laps over 24 races, zero censoring, refit on 2018-2023 and scored on 2024 - IPCW-Brier improves 0.19039 -> 0.16888, paired race-cluster bootstrap +0.0226 95% [+0.0073, +0.0353] with 200 of 200 draws improving and ~15x the family’s own 5-reseed noise floor; time-dependent AUC 0.6896 -> 0.7096; and the mean log bias moves -0.2741 -> +0.0680, so the level error CHANGES SIGN - the gauge stops over-predicting remaining tyre life (the dangerous direction for a number read as safe laps remaining) and starts under-predicting it slightly. WHAT IS NOT ESTABLISHED: the calibration slope, which moves 0.6736 -> 0.8754 but at paired-bootstrap P(improves) = 0.945 with an interval straddling zero - the fourth time 24 evaluation races have failed to resolve this quantity at 95%. THE CALIBRATION DEFECT IS NOT FIXED. Accuracy improves and the known dangerous bias direction is inverted; the calibration-slope gain remains statistically unresolved on the current 24-race evaluation set. THE COST, STATED: this family’s headline on this card is the mixture AFT NLL, and this change makes it WORSE - 1.99134 -> the value above, against a baseline that also moved (2.18868 -> 2.20099, because the AFT baseline is computed at the model’s own fitted scale), and
beats_baseline_significantfor this family flips True -> False. That is the expected direction and it was taken deliberately: work item 10c ruled the mixture the wrong headline for this target because it scores tyre-limit endings and exogenous SC/VSC/red interruptions in one likelihood, and these parameters were selected on the green-pit stratum instead. On that stratum’s own NLL the new parameters win (3.4377 -> 3.2922). Anyone reading this family’s headline as a quality signal is reading the metric the application does not use. Full evidence, all seven gates and the e-values:_improvements/work/10-competing-risks.mdunder 10e, artefacts in_improvements/eval/10e/. - NOT COMPARABLE TO v11. Work item 08m changed how
expected_compound_pace_sis formed (compound_cliff_severity, a fitted level shift, was being consumed as a per-lap rate, plus an unfitted 0.002*age^2 term). That column is subtracted intodriver_skill_residual_s, so the degradation trio’s targetnext_5_lap_cumulative_jump_sand the classifier’s labellaps_until_cliff_classare DIFFERENT QUANTITIES under unchanged column names: the target’s mean moved -1.8793 -> -0.3946 s, 10.79% of cliff labels changed class, and the trio’s training population moved 82,470 -> 81,619 rows. v11’s published headlines (p10 0.53202 / p50 1.04679 / p90 0.57851 pinball, cliff macro-F1 0.37187) were measured against the SUPERSEDED target. A smaller pinball loss here is a smaller target spread, not more skill - the non-leakage baseline moved too (p50 2.17766 -> 1.26289). NO IMPROVEMENT OVER v11 IS CLAIMED OR IMPLIED. The only admissible readings of this card are each model against its own baseline in this same run. The one exception is stint_life_regressor, whose target (remaining_stint_life_laps) 08m did not touch - only two of its 32 inputs changed meaning - though even there the eligible row set moved 121,193 -> 119,822, so it is a near-fixed rather than fixed comparison. - THERMAL-PROXY AND CROSS-SEASON LEAKAGE FIXES (work items 08e/08f, landed 2026-09-17; disclosure only - both fixes’ SQL predates this card’s numbers, so nothing here moved today). 08e:
int_lap_thermal_proxy’s stint baseline used to pool laps up toCEIL(stint_length_actual*0.60), reaching forward into laps not yet run (55.92% of mart rows, mean 4.65 laps) and invertingpush_residual’s sign against two targets; rebuilt as an expanding median over strictly-prior valid laps. Re-read on this card’s own v12 substrate: the rebuilt thermal family (push_residual+ the two cumulative push-load columns +surface_bulk_ratio) clears all five targets as an information gain, not leakage recovered - p10 +0.0665 pinball at 9.95x its own noise floor, p50 +0.0676 at 6.06x, p90 +0.0313 at 3.39x, cliff macro-F1 +0.0167 at 3.84x, stint-life AFT nloglik +0.0273 at 4.46x. 08f-1: the quantile trio’ssurvival_weight(an XGBoost sample weight, never a feature) used to pooltotal_per_compound/stints_reachingacross every ingested season, so a 2018 row’s weight was partly estimated from 2024; rebuilt as an expanding sum over strictly-prior seasons. Gated in isolation for the first time on this substrate: the weight vector changes substantially (93.3% of training rows), but every quantile headline delta stays inside its own 5-reseed noise floor (p10 0.49x, p50 0.15x, p90 -0.07x) - a real fix, a clean null on the headline;cliff_classifier/stint_life_regressornever consume this weight and are structurally unaffected. 08o (2026-09-21) then removed the weight from the training path altogether, so the sentence above about the other two families now holds for all five: uniform weights beat IPW on every head, though each headline delta sits INSIDE its own reseed floor (p10 0.70x, p50 0.06x, p90 0.41x) - the ruling rests on the permutation-null information cost at p10, which does clear its floor at -1.30x, and not on the headline.survival_weightis still built by the mart and still carried inIDENTIFIER_COLUMNS; it is now neither a feature nor a weight, and reaches no model. 08f-2:int_circuit_x_constructor_interaction’s two season-poolingGROUP BYs were rebuilt the same way. Proven algebraically, not merely measured within noise, to move zero live targets: its whole effect on a stint is a constant per (race, constructor), and every target this model is scored against differences laps within one stint, which cancels an exact constant. Full writeup:_improvements/eval/08e/and_improvements/eval/08f/. - The cliff classifier (macro-F1 ≈ 0.39 on 4-class cliff timing) is the weakest model by headline, and the headline is the wrong way to read it: 1.0 is not reachable on a label where laps inside a stint share a compound, a car, a circuit and a driver. Against an oracle handed the stint id and nothing else it reaches 61% of the ceiling. What limits it is not established. The label was the leading candidate and has been tested: laps_until_cliff_class is a first-crossing scan over int_compound_cliff_predicted, which is now bounded at source (compound_wear_max_s_per_lap). Repairing it moved 4.97% of rows and did not lift macro-F1, and a fresh 50-trial search on the repaired label did not beat the params tuned on the contaminated one. Neither the label nor the hyperparameters explain the number.
- Hyperparameters come from a 50-trial / season-fold Optuna search per target (ml/models/<target>_best_params.json);
make ml-retrainrefits at those params without re-searching. A target whose data has moved should be re-tuned, not just refit. - Skill is reported as a fraction of the ATTAINABLE quantity. A fraction above 1.0 means the model is not bounded by stint-level information, so no ceiling has been established for it - not that it is near-perfect.
- Every beats_baseline margin carries an interval over the season folds and over a stint-resampled bootstrap. The effective sample is ~7,100 stints, not ~137,000 laps.
- Cliff-onset priors are NULL for ~45% of laps (legacy compounds, un-fit circuits); XGBoost native-NaN carries them, documented rather than imputed.