Conceptual foundation: Methodology · feature source: fct_cliff_prediction_features.
Intended use
Race-strategy analysis and the Off the Pace web app (application layer): surfacing when a stint is about to fall off the degradation cliff and how much pace a driver is leaving on track. Not a betting or safety system.Models & headline performance
Every model is evaluated against a strong per-cohort baseline; all five beat it. Validation: season-grouped TimeSeriesSplit (expanding window, n_splits=5); whole seasons move together, the final fold validates on 2024.
Eval headline on the cv_final_fold (season 2024); CV is the season-grouped TimeSeriesSplit mean.
Of attainable is the column to read. A pinball of 0.20 or a macro-F1 of 0.40 says nothing on its own, because neither is a ratio against anything reachable — 1.0 is not a target any model of this data could hit. The column divides each headline’s improvement over an uninformed floor by the improvement a predictor with perfect stint-level knowledge could achieve.
A ⚑ marks a model that scores past that ceiling. It is not an error and not a broken metric: laps inside a stint share a compound, a car, a circuit, a fuel load and a driver, so a stint-level ceiling bounds only predictors that are constant within a stint. Clearing it is proof that the model is using within-stint variation — and that no ceiling has yet been established for it.
Per-model detail
degradation_regressor_p10
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.1) - Train rows: 81,619 · fit: 18.34s
- Artefacts:
ml/models/degradation_regressor_p10_v12.bst+ parity-testedml/models/degradation_regressor_p10_v12.onnx
degradation_regressor_p50
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.5) - Train rows: 81,619 · fit: 33.37s
- Artefacts:
ml/models/degradation_regressor_p50_v12.bst+ parity-testedml/models/degradation_regressor_p50_v12.onnx
degradation_regressor_p90
- Family / kind: degradation_regressor / quantile
- Objective:
reg:quantileerror(α=0.9) - Train rows: 81,619 · fit: 14.98s
- Artefacts:
ml/models/degradation_regressor_p90_v12.bst+ parity-testedml/models/degradation_regressor_p90_v12.onnx
cliff_classifier
- Family / kind: cliff_classifier / classification
- Objective:
multi:softprob - Train rows: 113,226 · fit: 88.9s
- Artefacts:
ml/models/cliff_classifier_v12.bst+ parity-testedml/models/cliff_classifier_v12.onnx
stint_life_regressor
- Family / kind: stint_life_regressor / survival
- Objective:
survival:aft - Train rows: 119,822 · fit: 9.74s
- Artefacts:
ml/models/stint_life_regressor_v12.bst+ parity-testedml/models/stint_life_regressor_v12.onnx
Features
32 features in 6 ablation groups. Categoricals are ordinal-encoded from the training map (NULL/unseen → −1).Leakage guards
driver_id and race_year are deliberately excluded: an adversarial probe recovers race_year from the remaining features at 0.997 accuracy (majority baseline 0.168)-constructor identity and compound generation encode the season, so race_year would be a backdoor. driver_id would let the trees relearn per-driver skill, the exact signal the mart strips via driver_skill_residual_s. Excluded (identity / target / leakage):circuit_key, drift_s_per_lap, driver_id, driver_skill_field_s, driver_skill_loro_mean_s, driver_skill_loro_s, driver_skill_proxy_s, driver_skill_residual_proxy_s, driver_skill_residual_s, is_training_eligible, lap_id, laps_until_cliff_class, next_3_lap_cumulative_jump_s, next_5_lap_cumulative_jump_s, next_lap_degradation_jump_detrended_s, next_lap_degradation_jump_s, race_id, race_year, remaining_stint_life_laps, stint_id, stint_length_laps, survival_weight
Validation
Baselines
Attainable ceilings
Skill is reported as a fraction of the ATTAINABLE quantity, not of 1.0. For the quantile trio the denominator is analytic — under a Gaussian shape a predictor that knew the stint could remove at most 1 - sqrt(1 - ICC) of the expected pinball, where ICC is the between-stint variance share estimated by one-way ANOVA. For the classifier it is a per-stint majority-class oracle; for stint life it is the censored AFT likelihood at a perfect prediction, which is a hard bound. A fraction above 1.0 means the model reaches past stint identity into within-stint variation.
Between-stint share is a one-way random-effects (ANOVA) estimate. The naive column is
var(per-stint means) / var(column), which counts within-stint scatter as between-stint signal at ~19 laps per stint; it is shown so the correction stays visible rather than silently applied.
Intervals on every claim
Every beats_baseline claim carries a paired-t interval over the season-grouped CV folds (n=5) plus a percentile bootstrap that resamples whole STINTS on the evaluation fold. Laps inside a stint share a compound, a car, a circuit, a fuel load and a driver, so a lap-grain interval understates the standard error; the ratio between the two is published per model rather than assumed.
Every
beats_baseline claim clears its own interval.
Effective sample: 6,203 stints over 81,619 laps (13.2 laps per stint). 0 stints straddle a season fold boundary, so the CV split was never the problem — every interval computed at lap grain was.
Calibration (quantile interval)
Nominal coverage 0.8 → raw [p10,p90] empirical 0.7851, split-conformal 0.8009 (mean interval width 5.9543s).Adversarial leakage probe
A throwaway model recoversrace_year from the features at 0.9965 vs a majority baseline of 0.1680-X carries strong residual temporal signal-race_year recoverable far above chance; race_year is correctly excluded from features.
Feature importance (SHAP vs permutation)
Underperforming cohorts
Surfaced, never dropped. Specific circuit/constructor cells where the model trails its non-leakage cohort-mean baseline; the model wins comfortably overall. 15 cohort cells lose to baseline (surfaced, never dropped):Reproducibility
- Seed:
20260528· dataset fingerprint:7852405cfa4abd47… - Libraries: xgboost 3.3.0, sklearn 1.9.0
- ONNX parity: all 5 boosters round-trip to ONNX within atol=1e-5 on a NaN-bearing sample (M2/M4)
- Reproduce:
make ml-all
Limitations
- NOT COMPARABLE TO v11. Work item 08m changed how
expected_compound_pace_sis formed (compound_cliff_severitywas fitted as a ~5.5-lap level shift and was being consumed as a per-lap rate, plus an unfitted 0.002*age^2 term). That column is subtracted intodriver_skill_residual_s, so the degradation trio’s targetnext_5_lap_cumulative_jump_sand the classifier’s labellaps_until_cliff_classare DIFFERENT QUANTITIES under unchanged column names: the target’s mean moved -1.8793 -> -0.3946 s, 10.79% of cliff labels changed class, and the trio’s training population moved 82,470 -> 81,619 rows. v11’s published headlines (p10 0.53202 / p50 1.04679 / p90 0.57851 pinball, cliff macro-F1 0.37187) were measured against the SUPERSEDED target. A smaller pinball loss here is a smaller target spread, not more skill - the non-leakage baseline moved too (p50 2.17766 -> 1.26289). NO IMPROVEMENT OVER v11 IS CLAIMED OR IMPLIED. The only admissible readings of this card are each model against its own baseline in this same run. The one exception is stint_life_regressor, whose target (remaining_stint_life_laps) 08m did not touch - only two of its 32 inputs changed meaning - though even there the eligible row set moved 121,193 -> 119,822, so it is a near-fixed rather than fixed comparison. - The cliff classifier (macro-F1 ≈ 0.35 on 4-class cliff timing) is the weakest model by headline, and the headline is the wrong way to read it: 1.0 is not reachable on a label where laps inside a stint share a compound, a car, a circuit and a driver. Against an oracle handed the stint id and nothing else it reaches 66% of the ceiling. What limits it is not established. The label was the leading candidate and has been tested: laps_until_cliff_class is a first-crossing scan over int_compound_cliff_predicted, which was unbounded until Phase 8 bounded it at source (compound_wear_max_s_per_lap). Repairing it moved 4.97% of rows and did not lift macro-F1, and a fresh 50-trial search on the repaired label did not beat the params tuned on the contaminated one. Neither the label nor the hyperparameters explain the number.
- Hyperparameters come from a 50-trial / season-fold Optuna search per target (ml/models/<target>_best_params.json);
make ml-retrainrefits at those params without re-searching. A target whose data has moved should be re-tuned, not just refit. - No live 2025 holdout yet-headline numbers are time-series CV until 2025 ingests.
- Skill is reported as a fraction of the ATTAINABLE quantity. A fraction above 1.0 means the model is not bounded by stint-level information, so no ceiling has been established for it - not that it is near-perfect.
- Every beats_baseline margin carries an interval over the season folds and over a stint-resampled bootstrap. The effective sample is ~7,100 stints, not ~137,000 laps.
- Cliff-onset priors are NULL for ~45% of laps (legacy compounds, un-fit circuits); XGBoost native-NaN carries them, documented rather than imputed.