Hyperparameter tuning: Optuna search, CV selection, and full refit

How Optuna TPESampler explores the 9-dimensional hyperparameter space for each of the five models, selects on season-grouped CV, and then refits on the full training set.

Every model in the pipeline is tuned by Optuna before its final fit. Tuning and training use deliberately separate data that separation is what keeps the headline CV metrics honest.

1. Search the space

   Optuna’s TPESampler (seeded at RANDOM_STATE) proposes hyperparameter configurations from a 9-dimensional space. MedianPruner kills unpromising trials after each CV fold, keeping the search efficient even with 50 trials.

   Hyperparameter	Type	Range	Role in the objective
   n_estimators	int	200–700	Tree count $K$ more trees = lower bias, more variance
   max_depth	int	3–8	Max depth per tree controls complexity
   learning_rate	float (log)	0.02–0.20	Shrinkage per tree lower = smoother, needs more trees
   subsample	float	0.6–1.0	Row sampling per tree regularisation via stochasticity
   colsample_bytree	float	0.6–1.0	Column sampling per tree feature-dropout regularisation
   min_child_weight	int	1–20	Minimum Hessian sum to make a leaf prunes fine-grained splits
   gamma ($\gamma$)	float (log)	1e-3–5.0	Minimum split-loss directly in the regularised objective
   reg_lambda ($\lambda$)	float (log)	1e-3–5.0	L2 leaf-weight regularisation
   reg_alpha ($\alpha$)	float (log)	1e-3–5.0	L1 leaf-weight regularisation

   The three regularisation strengths $\gamma, \lambda, \alpha$ map directly to the terms in the XGBoost regularised objective:

   $\Omega(f) = \gamma\,T \;+\; \tfrac{1}{2}\lambda\lVert w\rVert^2 \;+\; \alpha\lVert w\rVert_1$

   where $T$ is the leaf count and $w$ the leaf weights. All three are searched on a log scale because their effect is multiplicative a move from 0.001 to 0.01 matters as much as a move from 1.0 to 10.0.

2. Score on season-grouped CV

   Each trial is evaluated by season-grouped TimeSeriesSplit across five folds. The folds respect temporal ordering future seasons never inform models of past seasons and the headline metric varies by model family:

   • Quantile trio: mean pinball loss across folds (↓ lower is better)
   • Cliff classifier: mean macro-F1 across folds (↑ higher is better)
   • Stint-life regressor: mean RMSE across folds (↓ lower is better)

   After the study completes, the best parameters are written to ml/models/{target}_best_params.json. The Optuna study is persisted to ml/models/optuna_studies/{target}_{version}.db, so it is resumable a rerun with load_if_exists=True extends the existing study rather than starting from scratch.

3. Refit on the full training set

   Once the best hyperparameters are selected, the booster is trained on the entire training set (all seasons except the holdout). Cross-validation selected the configuration; the complete dataset trains the shipped model.

   This is the step that produces the .bst booster artefacts. The refit runs automatically after tuning completes make ml-tune chains into ml-train for the tuned version.

Reduced v1 budget. The shipped v1 parameters used a reduced tuning budget (15 trials / 3 folds / 30,000-row search subsample) for speed during initial development. The canonical make ml-tune (50 trials / 5 folds / full data) only improves the parameters. The beats-baseline gate holds regardless of budget.

Reproducible and resumable studies

Every study is keyed by {target}_{version} (e.g. cliff_classifier_v5) so a fresh tuning run for a new version gets its own namespace without overwriting the existing study. The TPESampler is seeded by RANDOM_STATE = 20260528, so two runs with the same data and the same budget produce the same parameter sequence.

The .db files are SQLite they can be inspected with optuna-dashboard or the scripts/inspect_trials.py utility. CI does not run ml-tune (it is too slow for pull requests); it runs ml-test against the committed .bst artefacts.

Relationships

Models

The five objectives and why each model’s loss function shapes what the tuning search is optimising.

Validation

Season-grouped CV the same scheme the tuning search uses as its objective.

Model Reference

The committed best-parameter values for each model, alongside the headline metrics they produce.

Pipeline

Where ml-tune sits in the full make ml-all DAG.