How well do the degradation models track what actually happened?

The shipped degradation models scored lap by lap against what happened predicted-vs-actual scatter, interval coverage, and a cliff-class confusion matrix for any ingested season.

A model that fits its training data proves nothing on its own. The blind-test scoreboard puts the shipped models’ predictions next to what actually happened, lap by lap, for the season you pick. It becomes a true blind test once the designated holdout season, 2026, starts to ingest. Until then every season it can show was part of the training set, so read it as in-sample fit, not as out-of-sample proof. Every number is read from the model artefacts, not hand-typed.

​

What it shows

Three panels score the models on the selected season:

• Predicted-vs-actual scatter: each point is one lap. The x-axis is the actual cumulative degradation jump over the next five laps (next_5_lap_cumulative_jump_s); the y-axis is the model’s p50 prediction. Points on the 45° diagonal are perfect calls. The p10 / p90 band is the 80% conformal envelope.
• Interval coverage rug: empirical coverage is the fraction of laps where the actual jump falls inside [p10, p90]. The target is 80% (conformal calibration from the training fold). Green bars hit; red bars miss.
• Cliff-class confusion matrix: predicted laps-until-cliff class (rows) versus actual class (columns), row-normalised so each cell’s intensity is the fraction of that predicted class landing in each actual bucket. The teal diagonal is correct; red off-diagonal is error.

All numbers are read from mart_degradation_predictions.parquet, joined to fct_cliff_prediction_features for the actuals and fct_stint_features for stint length. No values are hand-typed.

​

How to use it

1. Read the scatter’s spread around the diagonal tight is well-calibrated; a tilt means systematic over- or under-prediction.
2. Check the coverage rug against 80% consistent green means the intervals are honest on this season.
3. Read the confusion diagonal a strong teal diagonal means the cliff classifier rarely confuses adjacent classes.
4. Re-check after a season ingests once 2026 races arrive, they are scored by a model that never saw them.

​

Reading the signal correctly

The shipped v14 models were refit on every ingested season (2018–2025), so the predictions here are in-sample: they show how the models behave on laps they were trained on, which flatters both the scatter and the coverage. The honest out-of-sample numbers are the final cross-validation fold, 2025, scored by a model trained only on 2018–2024; those are on Model metrics. The conformal band targets 80% coverage by design, so slight over-coverage is expected even out of sample.

Nothing on this page is out-of-sample until 2026 data ingests. Even the 2025 cross-validation fold is a validation fold rather than a clean holdout, because it was also used to decide which features to admit.

​

Data source

The chart reads mart_degradation_predictions (the shipped models’ scored laps), joined to fct_cliff_prediction_features for actuals and fct_stint_features for stint length, via DuckDB-Wasm in the browser.

Column	Meaning
predicted_degradation_jump_p10_s / predicted_degradation_jump_s / predicted_degradation_jump_p90_s	Quantile predictions; p50 is the scatter y-axis, p10–p90 the band
actual_degradation_jump_s	Observed cumulative jump over the next five laps (the scatter x-axis), from next_5_lap_cumulative_jump_s
predicted_cliff_class / actual_cliff_class	Confusion-matrix rows and columns
predicted_remaining_stint_life_laps / actual_remaining_stint_life_laps	Stint-life prediction and the realised remainder (stint_length_laps - lap_in_stint)
is_in_envelope	Whether the lap is training-eligible (the same filter the models were fit on)

mart_degradation_predictions also carries is_holdout, which marks laps in the designated holdout season (2026). It is FALSE on every row until 2026 ingests.

For the full evaluation methodology, see the ML validation page.