Tests how to get an honest performance estimate: fold arithmetic, stratification, time-ordered and grouped splits, leakage through preprocessing, nested CV and fair model comparison.
Based on chapter 13. Every answer comes with an explanation and a link back to the section.
/Q13
Quiz
Question 1 of 26
Single-split variance and test-set selection bias Level 1
Progress
A team trains 50 models, scores each on the same test set, and reports the best score as the performance of the chosen model. Why is that number optimistic?
/02
练习From the book
7 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 13.1 [W]
Explain in your own words why choosing hyperparameters based on test-set performance invalidates that test set as an estimate of generalisation.
Show the model answer
Choosing hyperparameters based on test-set performance means the model was implicitly optimised, at least in part, to do well on that exact set of examples. The test set has stopped being purely "unseen data." It has become part of the selection process. Any resulting test-set score is now measuring how well the model fits that specific set, not how well it generalises to new data.
Exercise 13.2 [W]
State which CV strategy is correct for each: (a) a dataset of 30 rare-disease patient records; (b) daily sales figures for the past three years; (c) a roughly balanced 5,000-row classification dataset.
Show the model answer
(a) Leave-One-Out: 30 patient records is well under the "<50 examples" threshold for LOO. (b) TimeSeriesSplit: daily sales are strictly time-ordered, so random shuffling would leak the future into training. (c) Stratified k-fold (k=5): a roughly balanced, moderate-size dataset is the standard case Stratified k-fold with k=5 is designed for.
Exercise 13.3 [A]
Using Listing 13.6's pattern, compute the standard deviation of the ten seed-accuracy values (0.982, 0.991, 0.982, 0.974, 0.974, 0.982, 0.991, 0.965, 0.974, 1.000) and compare it to this chapter's 5-fold CV standard deviation (0.007). Which estimate would you trust more, and why?
Show the model answer
The ten seed-accuracy values have a standard deviation of about 0.010 (np.std), roughly 1 percentage point. That's somewhat larger than the 5-fold CV standard deviation of 0.007. The 5-fold CV estimate is still the more trustworthy of the two, and not just because its std happens to be smaller. It's built from five disjoint folds that together cover the whole dataset exactly once, systematically. The ten single-split values, by contrast, are ten separate 80/20 splits that overlap heavily. They aren't independent measurements of the same underlying quantity, not in a structured resampling sense. A single seed's score depends entirely on which specific examples happened to land in that one test set. CV's reported score is already an average over five such sets. That's exactly why it ends up both more stable and more representative than any pile of ad hoc reshuffles, however many are tried.
Exercise 13.4 [A]
Using Listing 13.4, explain what it would mean if the non-nested and nested CV estimates came back nearly identical (as this chapter's own 0.9543 vs. 0.9543 result did) versus if they came back several percentage points apart.
Show the model answer
A gap of zero, or effectively zero, between non-nested and nested estimates (like this chapter's 0.9543 vs. 0.9543 result) indicates the optimism bias is negligible for this dataset/search combination. That bias comes from selecting hyperparameters on the same data used to evaluate them. Either the dataset is easy enough to separate, or the hyperparameter search space is small enough, that overfitting to the inner CV folds isn't a serious problem here. A gap of several percentage points would indicate the opposite. The non-nested number would be meaningfully inflated by selection bias. Only the nested estimate should be reported or trusted as the honest expected performance.
Exercise 13.5 [E]
A teammate says "I ran 5-fold CV and got 94% ± 0.5%, so I'm confident in this number." Name one thing that CV std alone does not tell you about the estimate's reliability.
Show the model answer
CV std tells you how much the score varies across folds of this dataset with this model. It does not tell you whether the underlying data collection process itself is representative of the deployment population. It does not tell you whether there's target leakage baked into the features. It does not tell you whether the model will face distribution shift once deployed. A tight CV std is consistent with a reliable estimate on this data. But it says nothing about whether this data is the right data to be estimating from.
Exercise 13.6 [E]
A customer has a time-ordered dataset but their data science team used regular StratifiedKFold and reported a strong result. Using this chapter's Time-Series Cross-Validation section, write the two or three sentences you'd say about why that result should not be trusted as-is.
Show the model answer
"Regular StratifiedKFold on time-ordered data allows validation folds that sit chronologically before some of the training data within the same fold rotation. That means the model may have effectively seen future information relative to what it's being scored on. The strong reported result is very likely optimistic and would not hold up once the model only has access to past data at deployment time. I'd ask them to rerun with TimeSeriesSplit before trusting that number."
Exercise 13.7 [R]
Read the scikit-learn documentation's page on cross_validate versus cross_val_score. Identify one piece of information cross_validate returns that cross_val_score does not, and describe a situation from this chapter where that extra information would matter.
Show the model answer
Discussion only. cross_validate returns fit time and score time per fold (and can return train scores alongside validation scores, and multiple metrics simultaneously) that cross_val_score does not. This chapter's Procedure 13.5 (comparing two models) would benefit from cross_validate's per-fold timing when training cost, not just accuracy, is part of deciding between two otherwise similarly-accurate models.