Tests how to define search spaces, what grid, random and Bayesian search cost and when each fits, and why a tuned validation score needs an untouched test set.
Based on chapter 17. Every answer comes with an explanation and a link back to the section.
/Q17
Quiz
Question 1 of 23
Hyperparameters and search spaces Level 1
Progress
A customer trains an XGBoost classifier. Which of these is a hyperparameter rather than a learned parameter?
/02
练习From the book
7 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 17.1 [W]
Explain why grid search over 5 hyperparameters with 5 values each needs 55=3,125 training runs. Explain why that number would not shrink even if only 2 of the 5 hyperparameters mattered.
Show the model answer
55=3,125 because grid search enumerates every combination across all 5 hyperparameters' 5 values each, with no mechanism to notice or exploit the fact that only some of them matter. It treats every dimension as equally worth exhaustive coverage. Even if only 2 of the 5 affect performance, grid search still multiplies out all 5 dimensions blindly. It has no way to detect in advance which ones are the unimportant ones.
Exercise 17.2 [W]
State which search strategy is the reasonable default for: (a) tuning 2 hyperparameters on a model that trains in under a second; (b) tuning 8 hyperparameters on a neural network that takes 20 minutes per training run.
Show the model answer
(a) Grid search is reasonable: 2 hyperparameters and sub-second training keeps even exhaustive coverage cheap, and grid search's full, reproducible coverage has no real downside at this scale. (b) Bayesian optimisation (Optuna) is the better default: 8 hyperparameters and 20-minute training runs make each evaluation expensive, exactly the regime where sample efficiency (fewer wasted trials) matters most.
Exercise 17.3 [A]
Using Listing 17.5's measured results, compute the percentage-point gap between grid search's and random search's test AUC (0.8933 vs. 0.8942), and explain what about the grid's construction accounts for it.
Show the model answer
The gap is 0.8942−0.8933=0.0009 (0.09 percentage points) in random search's favour. It's small, but it isn't noise. It traces directly back to the grid's construction. With only two values per dimension across six hyperparameters, every one of grid search's 64 combinations sits at a corner of the search space: always the minimum or maximum subsample, max_depth, and so on. It can never place a trial in the interior of any dimension. Random search samples continuously across the same six ranges, so it can land on interior values the grid structurally excludes. It does: its winning configuration used max_depth=6 and subsample~0.88, values no combination in the 64-corner grid could ever produce.
Exercise 17.4 [A]
Using Procedure 17.4, sketch how you would add pruning to the Optuna study from this chapter's Worked Example if XGBoost's training reported intermediate validation AUC after every 10 boosting rounds, using optuna.pruners.MedianPruner.
Show the model answer
Sketch:
def objective(trial): params = {...} model = xgb.XGBClassifier( **params, random_state=42, verbosity=0 ) for round_num in range(0, params['n_estimators'], 10): # train incrementally / use eval_set with early rounds # validation AUC after this many rounds intermediate_auc = ... trial.report(intermediate_auc, step=round_num) if trial.should_prune(): raise optuna.exceptions.TrialPruned() return final_aucstudy = optuna.create_study( direction='maximize', pruner=optuna.pruners.MedianPruner( n_startup_trials=5, n_warmup_steps=10 ),)
Trials tracking below the median of previous trials at the same step get abandoned early, saving the remaining boosting rounds' compute for more promising configurations.
Exercise 17.5 [E]
A colleague argues Optuna should always be used "because it's the most sample-efficient." Using this chapter's measured trial counts and times (Optuna matched random search's final score in 5 trials and 5.7 seconds, but its full 64-trial run took 171.5 seconds against random search's 18.5), write the two or three sentences you'd say back about when that advice is right and when it isn't.
Show the model answer
"Optuna's sample efficiency pays off when you use it as a sample-efficient method: stop once it finds a good-enough configuration, rather than forcing it through the same trial budget as a cruder strategy. In this chapter's matched comparison, Optuna needed only 5 trials (5.7 seconds) to reach the CV AUC that random search needed its full 64 trials (18.5 seconds) to find. It went on to find the best score of the three strategies by the time its own 64 trials were up. But that full run took 171.5 seconds, more than the other two combined, because nobody told it to stop once it had already won. 'Always use Optuna' is right about its accuracy and its trial efficiency. It's wrong if the trial budget gets copied from a strategy that doesn't need to be stopped early."
Exercise 17.6 [E]
A team reports "we tuned hyperparameters and our validation AUC went from 0.91 to 0.97" as the headline result. Using this chapter's Validation Set Is Sacred section, write the question you'd ask before accepting that 0.97 figure.
Show the model answer
"Was that 0.97 validation AUC measured on the same folds used during the hyperparameter search, or on a held-out test set that never participated in tuning? If it's the former, the number is optimistic by construction: the search process selected for whichever configuration happened to score best on those specific folds, and the honest number is whatever a truly untouched test set reports."
Exercise 17.7 [R]
Read the Bergstra and Bengio (2012) abstract. Identify the specific dimensionality condition under which they show random search's advantage over grid search, and relate it to why this chapter's own six-hyperparameter, corners-only grid comparison showed the predicted gap.
Show the model answer
Discussion only. Bergstra and Bengio's argument is strongest in higher-dimensional search spaces, where only a small effective subset of hyperparameters meaningfully affects the objective, and grid search is additionally restricted to just a couple of values per dimension. The more total dimensions, and the coarser grid search's coverage of each one, the more its exhaustive-but-uniform corner sampling is wasted relative to a method that can reach the interior. This chapter's own comparison used six hyperparameters with a grid restricted to two boundary values each, exactly that regime. It showed the predicted gap. Random search beat grid on both CV and test AUC, and Optuna beat both by using its trial history to guide where it sampled next. It also showed a side of the argument Bergstra and Bengio don't dwell on: Bayesian optimisation's real advantage over random search shows up in trials needed, not in a matched-budget wall clock.