> ## Documentation Index
> Fetch the complete documentation index at: https://offthepace.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Cohorts surfaced, never dropped
> Every model is evaluated across compound, circuit, and rain-lap cohorts. Underperforming cells are recorded in the model card and published here never hidden.
Beyond the global headline, every model is scored across cohorts `compound`, `circuit_key`, and `is_rain_lap` dimensions. Cells where the model trails its per-cohort baseline are recorded in the model card and published here. The contract: **surface the losses, do not drop them**.
The underperforming cohorts below are honest; they are not a sign that the model fails overall. The per-cohort baseline is itself non-trivial (the cohort-mean degradation rate, a strong signal that "knows" the typical behaviour for this compound and circuit). Beating it everywhere is a high bar. The cells the model trails on are mostly low-n edge cases or circuits where the baseline's near-oracle advantage is structural.
## Underperforming cohort cells
The cell count comes from the model card it updates on every `make ml-card` run. The table below reflects the most recent evaluation.
| Dimension | Cohort | n | Model score | Baseline | Δ |
| ------------- | --------------------- | ----- | ----------- | -------- | ------ |
| `compound` | `_other` (p10) | 9 | 0.045 | 0.005 | −0.040 |
| `is_rain_lap` | `_other` (p10) | 15 | 0.544 | 0.534 | −0.010 |
| `compound` | `_other` (p50) | 9 | 0.081 | 0.016 | −0.066 |
| `is_rain_lap` | `_other` (p50) | 15 | 2.216 | 2.206 | −0.010 |
| `compound` | `_other` (p90) | 9 | 0.056 | 0.004 | −0.052 |
| `circuit_key` | Canadian GP | 843 | 0.136 | 0.126 | −0.010 |
| `compound` | `_other` (classifier) | 9 | 1.000 | 1.000 | 0 |
| `is_rain_lap` | `_other` (classifier) | 15 | 0.042 | 0.042 | 0 |
| `compound` | `_other` (stint-life) | 12 | 4.074 | 2.257 | −1.817 |
| `circuit_key` | Bahrain GP | 881 | 5.011 | 3.700 | −1.311 |
| `circuit_key` | British GP | 632 | 6.298 | 6.241 | −0.057 |
| `circuit_key` | Dutch GP | 1,208 | 8.803 | 8.322 | −0.481 |
| `circuit_key` | Japanese GP | 676 | 6.831 | 5.892 | −0.939 |
| `circuit_key` | Las Vegas GP | 692 | 8.815 | 5.296 | −3.520 |
*Model score and Baseline are the headline metric for each family: pinball ↓ for p10/p50/p90, macro-F1 ↑ for the classifier, RMSE ↓ for stint-life. Δ is signed so that negative always means the model trails, whichever direction its metric runs; a Δ of exactly 0 is a tie on a cell too small to separate them.*
## What the losses mean
**The `_other` compound cells (n = 9, 12).** The `_other` bucket covers legacy or unrecognised compound codes. With nine training rows, the baseline has nothing to learn it is essentially the global mean. These losses are structural artefacts of the small cell size rather than evidence the model has a problem with any real compound.
**The cliff classifier's two cells are ties, not losses.** Both are degenerate: nine `_other`-compound rows and fifteen rain laps, where model and baseline score identically. The classifier no longer trails on `INTERMEDIATE` compounds or at the Miami GP, which it did before the `laps_until_cliff_class` label was repaired.
**The `is_rain_lap` cells (n = 15 rain laps).** Fifteen laps in the evaluation set. A low-n artefact.
**Circuit-level losses (stint-life regressor).** The stint-life regressor's per-cohort baseline is **near-oracle**: it uses the true final stint length known only once the stint is over as its "prediction." On circuits where stint lengths are highly predictable (the Dutch GP historically runs long stints, the Las Vegas circuit tends toward specific strategic windows), that oracle baseline is genuinely hard to beat. The model wins globally (RMSE 7.73 vs 9.40) because the baseline is not available in real-time, but in the retrospective evaluation on those circuits the oracle wins. Las Vegas is the widest cell (8.82 vs 5.30).
The losses on Canadian GP (pinball) and on Bahrain, British and Japanese GPs (stint-life) are smaller, and mix sample-size effects with track-specific behaviour the model does not fully capture.
## The CI contract
`test_evaluate.py` asserts that cohort losses are **computed and surfaced** on every build, not that the model wins every cell. The gate is: "we measured this cohort, and if we lost, we know it." A model that silently skipped the cohort evaluation would fail the gate even if it was otherwise excellent.
See [Evaluation gates](/ml/ci/evaluation-gates) for the exact assertions.
## Relationships
The global validation context where the model wins overall.
The CI tests that enforce cohort surfacing and the beats-baseline contract.
The full cohort table from the latest model card updated on every `make ml-card` run.
The physics behind the cliff concept the classifier is trying to predict.