Hyperparameter Optimization: A Practical Guide

This guide is the opinionated "how to actually run a tuning job" companion to the

kit. It assumes you can already train a model and measure a validation metric;

the question is how to search the configuration space efficiently and trust the

result. Every code reference maps to a real symbol in src/hpo/.

The workflow in one picture

define metric  ->  define search space  ->  pick sampler + pruner/scheduler
      ->  set a budget  ->  run study  ->  inspect best + importance
      ->  widen/narrow space  ->  repeat until improvements plateau

Tuning is iterative. The first run is a reconnaissance pass with wide ranges; the

second run narrows toward the promising region. Two or three short rounds almost

always beat one giant run, because you spend later budget where it matters.

Step 1 — Nail down the objective

Before sampling a single configuration, decide:

• The metric. Use the metric you are actually judged on (e.g. balanced

accuracy, AUC, RMSE), not a proxy. The kit ships common scorers in

hpo.objective (accuracy, mse, rmse, mae, r2, neg_mse); pass any

callable (y_true, y_pred) -> float for anything custom.

• The direction. "maximize" for scores, "minimize" for losses. Keep it

consistent across the space, the pruner, and the study. The neg_* scorers

exist so a loss can be maximized uniformly when that is more convenient.

• The estimator of generalization. A single train/validation split is noisy

and easy to overfit through hundreds of trials. Prefer K-fold

cross-validation. hpo.objective.Objective wraps K-fold CV for you and returns

the mean fold score.

Pitfall — tuning on the test set. Never let the test set influence the

search. Tune against CV folds or a dedicated validation split, and evaluate the

single chosen configuration on the held-out test set exactly once.

Step 2 — Choose a sampler

The sampler decides *which configuration to try next*. The kit exposes the Optuna

samplers through StudyConfig(sampler=...):

Rule of thumb: start with random (or qmc) for one round to map the

terrain, then switch to tpe to exploit it. If a smart sampler cannot beat random

search by a meaningful margin, your space or metric is the problem, not the

sampler. The bundled examples/random_search_baseline.py gives you that baseline

with zero dependencies.

Step 3 — Prune the losers

Most configurations are mediocre and reveal themselves early. Pruning stops their

training once the intermediate learning curve makes the outcome obvious, which can

multiply throughput several-fold. Pruning needs two things:

1. an objective that reports intermediate values (per epoch, per boosting

round, or — as in hpo.objective.Objective — per CV fold); and

2. a pruner that compares the running trial against completed ones.

The kit implements the decision logic in pure Python so you can reason about it:

• MedianPruner — prune a trial whose best-so-far value is worse than the

median of completed trials at the same step. The sensible default.

• PercentilePruner(percentile=p) — keep the best p%, prune the rest.

MedianPruner is exactly PercentilePruner(50). Lower p is more aggressive.

• ThresholdPruner(lower=..., upper=...) — a guard rail: abort trials that

cross an absolute bound (e.g. a diverged loss), independent of other trials.

• EarlyStopping(patience=...) — classic in-loop stopping for a single trial;

no cross-trial baseline required.

Two parameters control pruning aggressiveness:

• n_startup_trials — do not prune until this many trials have *completed*, so the

baseline distribution is meaningful. Too low and you prune against noise.

• n_warmup_steps — do not prune any trial before this step, so slow-starting but

ultimately-strong configurations are not killed prematurely. Models with warm-up

schedules or heavy regularization need a larger value here.

Convert any of these to the real Optuna pruner with to_optuna_pruner(...), or

just hand the kit pruner to StudyConfig(pruner=...) and run_study does it.

Step 4 — Schedule trials (Ray Tune)

Ray Tune reframes pruning as *scheduling* across many parallel trials. The kit's

TuneConfig(scheduler=...) selects:

• ASHA (asha) — Async Successive Halving. Runs many trials for a short

grace_period, then promotes the top 1/reduction_factor to train longer,

repeating up to max_t. The workhorse for parallel HPO; the Ray analogue of

median/percentile pruning.

• PBT (pbt) — Population Based Training. Maintains a population, periodically

copying weights from strong trials and perturbing their hyperparameters. Powerful

for schedules that should *change during training* (e.g. learning rate). Requires

scheduler_kwargs["hyperparam_mutations"].

• Median stopping (median) — the simplest comparative rule, a good first try.

grace_period, reduction_factor, and max_t trade exploration for speed:

larger grace_period is safer but slower; larger reduction_factor is faster but

riskier. See examples/ray_tune_torch.py for a complete ASHA run.

Step 5 — Set a budget (and persist it)

• Trial count. A practical default is 20–30 trials per tunable dimension, then

inspect convergence. The optimization_history helper returns the running best,

which is the convergence curve — if it has been flat for many trials, stop.

• Parallelism. random/qmc/ASHA parallelize cleanly; tpe is partly

sequential (later trials depend on earlier results) but still benefits from

modest parallelism.

• Persistence and resume. Set `StudyConfig(storage="sqlite:///hpo.db",

study_name=..., load_if_exists=True)`. You can stop and resume, or point several

worker processes at the same database to fan trials out. examples/optuna_sklearn.py

does exactly this — re-run it and it continues the previous study.

Step 6 — Multi-objective optimization

When you care about more than one metric (accuracy *and* latency, recall *and*

precision), there is no single "best" — there is a Pareto front of

non-dominated trade-offs. To run it:

• pass a list of scorers and matching directions: `Objective(..., scorer=["accuracy",

my_latency_scorer]) with directions=["maximize", "minimize"]` on the study;

• the objective returns a tuple, and Optuna records study.best_trials (the front);
• pareto_front(records, directions) recomputes the front from backend-agnostic

records, and you pick the trade-off that fits your product constraints.

Do not collapse multiple objectives into a hand-weighted sum unless the weights are

genuinely known — the weighting hides the trade-off you are trying to see.

Step 7 — Read the results

After a study finishes, from_optuna_study(study) gives you TrialRecords that

feed every analysis helper:

• best_trial / top_k — the winners and the runners-up (inspect the top few:

if they cluster, you have found a stable region).

• param_importance — a fast variance-reduction proxy telling you which knobs

moved the metric. Use it to redesign the next round: widen important params,

freeze unimportant ones at a good value to cut dimensionality.

• optimization_history — the convergence curve.
• export_csv / export_json — dump everything for a notebook or a report.

Common failure modes

• Best value sits on a range boundary. The optimum is probably outside your

range — widen it and re-run. See search-space-design.md.

• Pruning too aggressively. If completed-trial counts are tiny and many trials

are pruned, raise n_startup_trials and n_warmup_steps.

• Noisy single-split scoring. Move to K-fold CV; the variance between splits is

often larger than the differences you are chasing.

• Over-tuning. Hundreds of trials against one validation split will find a

configuration that is lucky, not good. Cross-validate and keep an untouched test

set.

• Ignoring seeds. Set seed on the sampler and CV so results are reproducible

and comparable across rounds.

The short version: baseline with random search, exploit with TPE/ASHA, prune the

losers, cross-validate, inspect importance, narrow the space, repeat.

Search Space Design

The search space is the highest-leverage decision in hyperparameter optimization.

A great sampler cannot rescue a badly shaped space, and a well-shaped space makes

even random search competitive. This guide covers the design rules and shows how

to express them with the kit's declarative SearchSpace.

Why a declarative space

The kit defines one space that compiles to both backends:

from hpo import SearchSpace, FloatParameter, IntParameter, CategoricalParameter

space = SearchSpace([
    FloatParameter("learning_rate", 1e-4, 1e-1, log=True),
    IntParameter("n_estimators", 50, 500, log=True),
    IntParameter("max_depth", 2, 12),
    CategoricalParameter("booster", ["gbtree", "dart"]),
])

space.suggest(trial)   # -> Optuna (or RandomTrial) values
space.to_ray()         # -> Ray Tune param_space
space.sample(seed=0)   # -> stdlib random draw

Defining the space once is not a convenience detail — it removes the most common

source of irreproducible HPO: the Optuna study and the Ray Tune param_space

silently drifting apart so their results are no longer comparable.

Rule 1 — Log scale for multiplicative parameters

Some parameters are *additive* (a step of +1 means the same thing everywhere) and

some are *multiplicative* (what matters is the ratio, e.g. 0.001 → 0.01 is the same

"distance" as 0.01 → 0.1). Sample multiplicative parameters on a log scale so

the sampler spends equal effort per order of magnitude.

Use log=True for:

• Learning rates — FloatParameter("lr", 1e-5, 1e-1, log=True)
• Regularization strengths — L1/L2 penalties, weight_decay, alpha, C
• Counts spanning orders of magnitude — IntParameter("n_estimators", 50, 2000, log=True)

Use linear (the default) for bounded, additive parameters: dropout in [0, 0.5],

subsample in [0.5, 1.0], max_depth in [2, 12].

A linear search over lr in [1e-5, 1e-1] wastes ~90% of its samples in

[1e-2, 1e-1] and almost never tries the small values that often win. log=True

fixes this. The kit enforces low > 0 for log floats and low >= 1 for log ints.

Rule 2 — Start wide, then narrow

Round one should be deliberately generous: you are locating the promising region,

not polishing it. Round two narrows around the winners.

# Round 1 — reconnaissance
SearchSpace([FloatParameter("lr", 1e-5, 1.0, log=True),
             IntParameter("max_depth", 1, 16)])

# Round 2 — exploitation (best was lr≈3e-3, depth≈6)
SearchSpace([FloatParameter("lr", 1e-3, 1e-2, log=True),
             IntParameter("max_depth", 4, 8)])

Drive the narrowing with data: param_importance tells you which parameters

deserve a wider second-round range, and top_k shows where the good configurations

cluster.

Rule 3 — Watch for boundary pile-up

After a round, check the best configuration against your bounds. **If the optimum

sits on an edge of the range, the true optimum is probably outside it.** Widen that

side and re-run. A space whose winner is glued to max_depth = 16 (your upper

bound) is telling you to try max_depth = 24.

This single check catches more bad spaces than any other diagnostic.

Rule 4 — Pick the right parameter type

• FloatParameter — genuinely continuous quantities (rates, fractions,

temperatures). Add step= only when the model truly discretizes the value.

• IntParameter — counts and depths. Use log=True for counts that span

orders of magnitude; step= for coarse grids (e.g. step=16 for layer widths).

• CategoricalParameter — unordered choices (kernel, optimizer, activation).

A subtle but important distinction: do not encode an ordered numeric quantity

as a categorical just to restrict it. `CategoricalParameter("hidden", [64, 128,

256])` throws away the ordering a sampler could exploit. Prefer

IntParameter("hidden", 64, 256, step=64) — unless the values are genuinely

unordered, in which case categorical is correct.

Rule 5 — Reduce dimensionality you don't need

Every added dimension dilutes a fixed trial budget. Two tactics:

• Freeze the unimportant. After a round, set low-importance parameters to a good

constant and drop them from the space. The remaining budget concentrates on what

matters.

• Tie correlated parameters. If two parameters always move together, search one

and derive the other in build_model.

A focused four-parameter space with 60 trials usually beats a sprawling twelve

-parameter space with the same 60 trials.

Rule 6 — Conditional and hierarchical parameters

Some parameters only matter for certain choices — degree is meaningless unless the

SVM kernel is "poly". There are two clean ways to express this with the kit:

A. Branch inside build_model. Sample everything in the flat space and ignore

irrelevant values when constructing the model:

space = SearchSpace([
    CategoricalParameter("kernel", ["rbf", "poly", "linear"]),
    FloatParameter("C", 1e-2, 1e2, log=True),
    IntParameter("degree", 2, 5),     # only used when kernel == "poly"
    FloatParameter("gamma", 1e-4, 1e1, log=True),  # ignored when kernel == "linear"
])

def build_model(params):
    from sklearn.svm import SVC
    kwargs = {"C": params["C"], "kernel": params["kernel"]}
    if params["kernel"] == "poly":
        kwargs["degree"] = params["degree"]
    if params["kernel"] != "linear":
        kwargs["gamma"] = params["gamma"]
    return SVC(**kwargs)

B. Conditional suggestion (Optuna). Optuna supports define-by-run spaces where

later suggest_* calls depend on earlier ones. Write a custom objective that calls

trial.suggest_categorical("kernel", ...) and only suggests degree in the poly

branch. Use this when the irrelevant dimensions would otherwise confuse the

sampler's model; the flat approach (A) is simpler and fine for small trees of

conditions.

Rule 7 — Load spaces from config

SearchSpace.from_dict builds a space from plain dictionaries, so spaces can live

in JSON or YAML alongside your experiment config and be versioned independently of

code:

spec = {
    "learning_rate": {"type": "float", "low": 1e-4, "high": 1e-1, "log": True},
    "n_estimators":  {"type": "int",   "low": 50,   "high": 500,  "log": True},
    "max_depth":     {"type": "int",   "low": 2,    "high": 12},
    "max_features":  {"type": "categorical", "choices": ["sqrt", "log2", null]},
}
space = SearchSpace.from_dict(spec)

space.describe() is the inverse — it returns exactly this dict for any space, so

you can snapshot the space that produced a result into your experiment log.

Worked starting points

These are sensible *first-round* spaces. Widen or narrow per the rules above.

Gradient-boosted trees (XGBoost / LightGBM / sklearn):

SearchSpace([
    IntParameter("n_estimators", 100, 2000, log=True),
    FloatParameter("learning_rate", 1e-3, 3e-1, log=True),
    IntParameter("max_depth", 3, 12),
    FloatParameter("subsample", 0.5, 1.0),
    FloatParameter("colsample_bytree", 0.5, 1.0),
    FloatParameter("reg_lambda", 1e-3, 1e2, log=True),
])

Neural network (MLP / CNN):

SearchSpace([
    FloatParameter("lr", 1e-4, 1e-1, log=True),
    FloatParameter("weight_decay", 1e-6, 1e-2, log=True),
    IntParameter("hidden_size", 64, 512, step=64),
    IntParameter("num_layers", 1, 5),
    FloatParameter("dropout", 0.0, 0.5),
    CategoricalParameter("optimizer", ["adam", "adamw", "sgd"]),
])

Support Vector Machine:

SearchSpace([
    FloatParameter("C", 1e-2, 1e3, log=True),
    FloatParameter("gamma", 1e-4, 1e1, log=True),
    CategoricalParameter("kernel", ["rbf", "poly", "sigmoid"]),
])

Checklist before you launch

• [ ] Multiplicative parameters use log=True.
• [ ] Ranges are wide enough that the optimum is unlikely to sit on a boundary.
• [ ] Ordered numerics use Int/Float, not Categorical.
• [ ] The space has only the dimensions you can afford to search.
• [ ] Conditional parameters are handled (branch in build_model or define-by-run).
• [ ] The space is captured (describe() / from_dict) for reproducibility.

Get the space right and the rest of HPO becomes routine.