◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Chapter 13

Cross-Validation & Model Selection

A model trained and evaluated on a single 80/20 split reports one number. Let’s say 94% accuracy. That number is less trustworthy than it looks. A different random split could easily give 91% or 97%. There’s no way to tell from a single split which one you got. Now say twenty candidate models are compared on the same test set, and the best one is reported. That number is optimistic by construction. The best of twenty random-ish outcomes is not a neutral estimate. This chapter covers the systematic fix: cross-validation. It also covers the ways teams still leak information past it, even when they think they’re doing it right.

13.1 Motivation

The basic rule: any data used to make decisions about a model cannot also be used to evaluate it. Choosing hyperparameters, selecting features, or picking between algorithms based on a test set uses that test set up. It’s no longer a clean estimate of unseen-data performance. Cross-validation solves this. It repeatedly splits the data into different train/validation partitions and averages the results. No single split is “the answer”. There is no best. There is only the distribution of performance across many splits.

13.2 Exposition

13.2.1 The Problem With a Single Train/Test Split

An 80/20 split on a 569-example dataset leaves only 114 examples to test on. That’s enough for the estimate to swing a lot, depending on which 114 happened to land in the test set. With a small-to-moderate dataset, the standard error of a single accuracy estimate is large. A model can look like the best or worst result purely by chance.

13.2.2 k-Fold Cross-Validation

Split the data into kk equal-sized folds. Train on k−1k-1 folds. Validate on the remaining fold. Rotate which fold is held out. After kk rounds every example has been in the validation set exactly once. Average the kk scores. This uses every example for both training and validation. The spread across folds, not just the mean, tells you how stable the model is. A mean with tight fold agreement is a different finding from the same mean with folds bouncing around. k=5k=5 is the standard default. k=10k=10 gives slightly lower bias (more training data per fold) at the cost of more compute. k=nk=n, where nn is the total number of examples in the dataset, is leave-one-out cross-validation (LOOCV). Each fold holds out exactly one example, trains on the remaining n−1n-1, and tests on that single point. Repeat nn times, once per example, and average. LOOCV is unbiased in one sense: every training set is as large as it can possibly be. But it means fitting the model nn separate times. And each fold’s score is a single right-or-wrong outcome, with no averaging within the fold itself. That’s where the Trade-offs table’s “high variance between folds” comes from. LOOCV is practical for the tiny datasets it’s usually reserved for. Fitting a model 500 times, leaving out one of 500 rows each time, is a very different thing on a larger dataset.

13.2.3 Stratified k-Fold

Regular k-fold can, by chance, produce folds with very different class ratios. If 5% of the data is the positive class, one fold could land at 0% positives, making that fold’s score incomparable to the others. Stratified k-fold preserves the class ratio in every fold, and is the correct default for classification; cross_val_score uses StratifiedKFold automatically when the estimator is a classifier.

13.2.4 Time-Series Cross-Validation

Random shuffling is wrong for time-ordered data. Training on next week’s data to predict this week’s leaks future information into the training set. The result is an optimistic estimate that collapses the moment it meets production. TimeSeriesSplit enforces time order. The validation set is always in the future relative to its training set. Training windows grow across folds. Validation windows stay strictly ahead of them. This is a strict requirement for any time-ordered dataset: sales, prices, sensor readings, user behaviour over time.

13.2.5 The Overfitting-to-Test-Set Problem

A common, easy-to-fall-into failure mode. Train fifty different models. Measure each on the same test set. Report the best one’s test accuracy as “the” model’s performance. Selecting the maximum over fifty imperfect estimates introduces selection bias. The winning number is optimistic. Not because the winning model is secretly worse, but because the selection process itself systematically favours whichever model got lucky on that specific test set.

13.2.6 Nested Cross-Validation

Nested CV separates model selection from model evaluation using two loops. An outer loop (typically k=5k=5) estimates generalisation performance. An inner loop (typically k=3k=3, runs within each outer training set) selects the best hyperparameters. For each outer fold, run the inner-loop hyperparameter search on that fold’s training data. Then evaluate the resulting best model on that fold’s held-out test data, a test set the inner search never touched. The hyperparameters chosen may differ across outer folds. That’s expected. The outer loop is estimating expected performance. It is not selecting the model to ship.

13.3 Trade-offs and Limits

CV Strategy When to use Weakness
Single train/test split Quick sanity checks only High variance, unreliable
k-Fold (k=5) Standard default for most tasks May lose temporal ordering
Stratified k-Fold Classification with class imbalance Same as k-Fold otherwise
TimeSeriesSplit Any time-ordered data Training set smaller in early folds
Nested CV Hyperparameter tuning + honest evaluation Computationally expensive
Leave-One-Out Tiny datasets (<50 examples) Slow; high variance between folds

13.4 Procedures

Procedure 13.1 (Run k-Fold CV in One Line). When to use: estimating generalisation performance of any model.

Listing 13.1: Standard k-fold CV inside a Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring='accuracy')
print(f'Accuracy: {scores.mean():.3f} ± {scores.std():.3f}')

Common mistake: fitting a scaler (or any preprocessing) on the whole dataset before cross-validating, rather than inside a Pipeline. That leaks validation-fold statistics into training and produces optimistic scores. Always put preprocessing inside the pipeline being cross-validated.

Procedure 13.2 (Use Stratified k-Fold Explicitly). When to use: imbalanced classification, or when explicit control over the CV strategy is needed.

Listing 13.2: Explicit stratified k-fold
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
)

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
    pipe, X, y, cv=skf, scoring='f1_weighted'
)

Procedure 13.3 (Set Up Time-Series CV). When to use: any dataset where rows are time-ordered.

Listing 13.3: TimeSeriesSplit: validation always in the future
from sklearn.model_selection import (
    TimeSeriesSplit,
    cross_val_score,
)

# Verify chronological order first
df = df.sort_values('date').reset_index(drop=True)
tss = TimeSeriesSplit(n_splits=5)
scores = cross_val_score(
    pipe, X_ts, y_ts, cv=tss, scoring='accuracy'
)

Procedure 13.4 (Implement Nested CV for Honest Hyperparameter Tuning). When to use: tuning hyperparameters and reporting a valid performance estimate at the same time.

Listing 13.4: Nested cross-validation: inner search, outer honest evaluation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (
    GridSearchCV,
    cross_val_score,
    StratifiedKFold,
)

rf_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', RandomForestClassifier(random_state=42)),
])

inner_cv = StratifiedKFold(
    n_splits=3, shuffle=True, random_state=42
)
outer_cv = StratifiedKFold(
    n_splits=5, shuffle=True, random_state=42
)

param_grid = {
    'clf__n_estimators': [50, 100, 200],
    'clf__max_depth': [3, 5, None],
}
search = GridSearchCV(
    rf_pipe, param_grid, cv=inner_cv, scoring='accuracy', n_jobs=-1
)

nested_scores = cross_val_score(
    search, X, y, cv=outer_cv, scoring='accuracy'
)
mean, std = nested_scores.mean(), nested_scores.std()
print(f'Nested CV: {mean:.3f} ± {std:.3f}')

# After nested CV establishes the honest estimate, refit
# on all data for the final model:
search.fit(X, y)
final_model = search.best_estimator_

Procedure 13.5 (Compare Two Models Properly). When to use: before claiming “Model A is better than Model B.”

Listing 13.5: Paired t-test on cross-validation fold scores
from scipy import stats

scores_a = cross_val_score(
    model_a, X, y, cv=10, scoring='accuracy'
)
scores_b = cross_val_score(
    model_b, X, y, cv=10, scoring='accuracy'
)
t_stat, p_val = stats.ttest_rel(scores_a, scores_b)

Common mistake: comparing two models’ mean CV scores directly without a significance test. A 0.3 percentage-point difference in means can easily be noise. A paired t-test on matched fold scores is the quick check. Treat it as optimistic: the folds share training data, so their scores aren’t independent, and a plain t-test overstates confidence. Nadeau and Bengio’s corrected resampled t-test adjusts for that.

13.5 Worked Example

Take the Breast Cancer Wisconsin dataset: 569 examples, binary classification. Run the same logistic-regression pipeline through a fresh 80/20 split under ten different random seeds. Accuracy ranges from 0.965 to 1.000, a 3.5 percentage-point spread from nothing but which examples happened to land in the test set:

Listing 13.6: Ten random splits, same model, same data: different reported accuracy
for seed in range(10):
    X_tr, X_te, y_tr, y_te = train_test_split(
        X, y, test_size=0.2, random_state=seed, stratify=y
    )
    pipe.fit(X_tr, y_tr)
    acc = accuracy_score(y_te, pipe.predict(X_te))
    print(f'Seed {seed}: {acc:.3f}')

Any single one of those ten numbers, reported alone, would have looked like a clean, confident result. Run a 5-fold stratified CV instead, on the same pipeline and data. It reports fold scores of [0.982, 0.982, 0.974, 0.974, 0.991], mean 0.981, std 0.007. That’s a single number with an honest error bar attached, built from the same data used five different ways rather than one lucky (or unlucky) way.

Nested CV tunes a Random Forest’s n_estimators/max_depth via a 3-fold inner search inside a 5-fold outer loop. The result here is more subtle than the “nested is always dramatically lower” framing sometimes suggests. The non-nested estimate (the search’s own best_score_, from the same folds used to pick the winning hyperparameters) reports 0.9543. The honest nested estimate reports 0.9543 too. That’s a gap too small to separate from rounding on this particular dataset, not the large optimism bias the Overfitting-to-Test-Set Problem describes in the worst case. That’s still a real, useful finding in its own right. The size of the nested-vs-non-nested gap depends on how easy the dataset is to separate and how much hyperparameter search space was explored. The only way to know whether a given case is a small gap or a large one is to run nested CV, not to assume.

AWS SageMaker Equivalent
# SageMaker Automatic Model Tuning runs the equivalent of a
# hyperparameter search with built-in objective-metric
# tracking across training jobs
import sagemaker
from sagemaker.tuner import (
    HyperparameterTuner,
    IntegerParameter,
)

tuner = HyperparameterTuner(
    estimator=xgb_estimator,
    objective_metric_name="validation:accuracy",
    hyperparameter_ranges={
        "max_depth": IntegerParameter(3, 10)
    },
    max_jobs=20,
    max_parallel_jobs=4,
)
tuner.fit({
    "train": train_s3_uri,
    "validation": val_s3_uri,
})

When to prefer SageMaker Automatic Model Tuning: when the search needs to run as many independent, parallel managed training jobs rather than in-process GridSearchCV. Useful once a single search no longer fits comfortably on one machine, or when the team wants tuning history tracked centrally rather than in a notebook.

13.6 History and Further Reading

Thakur, Ch. 2, gives practical cross-validation patterns with code (Thakur 2020). Murphy’s Introduction, sections 4.5-4.6, covers the statistical foundations of model selection (Murphy 2022). Burkov, Ch. 5, covers hyperparameter tuning and model selection (Burkov 2019).

13.7 How This Connects

This chapter builds on Chapter 11 (a model is needed before it can be cross-validated). It looks ahead to Chapter 16 for the evaluation metrics used within CV. It enables Chapter 17’s hyperparameter optimisation, which is always done via CV. It’s needed for essentially every model-comparison decision in the rest of this book. It’s often confused with test-set evaluation. CV is for model development and selection. The test set is for one final evaluation, after all decisions have already been made. Touch the test set once.

13.8 Exercises

  1. [W] Explain in your own words why choosing hyperparameters based on test-set performance invalidates that test set as an estimate of generalisation.

  2. [W] State which CV strategy is correct for each: (a) a dataset of 30 rare-disease patient records; (b) daily sales figures for the past three years; (c) a roughly balanced 5,000-row classification dataset.

  3. [A] Using Listing 13.6’s pattern, compute the standard deviation of the ten seed-accuracy values (0.982, 0.991, 0.982, 0.974, 0.974, 0.982, 0.991, 0.965, 0.974, 1.000) and compare it to this chapter’s 5-fold CV standard deviation (0.007). Which estimate would you trust more, and why?

  4. [A] Using Listing 13.4, explain what it would mean if the non-nested and nested CV estimates came back nearly identical (as this chapter’s own 0.9543 vs. 0.9543 result did) versus if they came back several percentage points apart.

  5. [E] A teammate says “I ran 5-fold CV and got 94% ± 0.5%, so I’m confident in this number.” Name one thing that CV std alone does not tell you about the estimate’s reliability.

  6. [E] A customer has a time-ordered dataset but their data science team used regular StratifiedKFold and reported a strong result. Using this chapter’s Time-Series Cross-Validation section, write the two or three sentences you’d say about why that result should not be trusted as-is.

  7. [R] Read the scikit-learn documentation’s page on cross_validate versus cross_val_score. Identify one piece of information cross_validate returns that cross_val_score does not, and describe a situation from this chapter where that extra information would matter.

13.9 Mental Model

Cross-validation is not a model selection technique. It’s an estimation technique. It tells you how well your model type generalises. The test set tells you how well this specific fitted model generalises. Use CV during development. Touch the test set once, at the end, to report the final result. Never let it take part in any decision along the way.

Burkov, Andriy. 2019. The Hundred-Page Machine Learning Book. Andriy Burkov.
Murphy, Kevin P. 2022. Probabilistic Machine Learning: An Introduction. MIT Press.
Thakur, Abhishek. 2020. Approaching (Almost) Any Machine Learning Problem. Abhishek Thakur.
online /books/classical-data-science/ch13-cross-validation-model-selection Newcastle, UK v0.1.0 // rss