Tests the bias-variance trade-off and the inductive bias of each major algorithm family: linear models, SVMs, trees, KNN and Naive Bayes. Includes hand calculations and customer scenarios on choosing a first model.
Based on chapter 11. Every answer comes with an explanation and a link back to the section.
/Q11
Quiz
Question 1 of 25
Bias, variance and the train/test gap Level 1
Progress
In Expected Test Error=Bias2+Variance+Irreducible Error, which term can no choice of model or amount of data remove?
/02
练习From the book
7 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 11.1 [W]
For each of the following, name the algorithm family (Linear, Support Vector Machine, Tree-Based, Distance-Based, Probabilistic) and state its core assumption in one sentence: (a) spam detection using word-count features; (b) predicting house price from square footage and location; (c) classifying flowers from petal measurements using their five closest labelled neighbours.
Show the model answer
(a) Spam detection using word-count features: Probabilistic (Naive Bayes). Its core assumption is that features are conditionally independent given the class, which is a reasonable approximation for bag-of-words counts given a topic. (b) Predicting house price from square footage and location: Linear. Its core assumption is that the target is well-approximated by a linear combination of the input features. (c) Classifying flowers from petal measurements using their five closest labelled neighbours: Distance-Based (KNN). Its core assumption is that nearby points in feature space share a label, so a new point's class is decided by a majority vote among its closest training examples.
Exercise 11.2 [W]
Explain, in your own words, why increasing model flexibility (e.g., letting a decision tree grow deeper) tends to reduce bias while increasing variance. Use the bias-variance equation from this chapter.
Show the model answer
The equation is Expected Test Error=Bias2+Variance+Irreducible Error. A shallow tree can only express a small number of coarse if/else rules, so it's forced to represent the true decision boundary poorly no matter what training data it sees. That's high, stable bias. Letting the tree grow deeper adds more splits, so it can carve out increasingly fine-grained regions that match the true boundary more closely, which is exactly what lowers the bias term. But those extra splits are fit to the specific noise in the training sample, and a different training set would produce different deep splits, so the fitted function becomes unstable across resamples, which is exactly what raises the variance term. Flexibility is a dial on how much the model is allowed to bend to fit what it's shown. Bending further to reduce systematic error is the same mechanism that makes the fit more sensitive to which particular examples it saw.
Exercise 11.3 [A]
Using the benchmark table in the Worked Example, identify every algorithm whose test accuracy exceeds its CV mean accuracy. Explain what that ordering does, or doesn't, tell you about overfitting for those models.
Show the model answer
Four algorithms in the benchmark table show test accuracy exceeding CV mean accuracy: Logistic Regression (0.982 vs. 0.980, +0.002), SVM (0.982 vs. 0.971, +0.011), Random Forest (0.956 vs. 0.954, +0.002), and Decision Tree (0.912 vs. 0.910, +0.002). Three of those four gaps are tiny and identical, +0.002, negligible next to the CV standard deviations reported alongside them (±0.013 for Logistic Regression, ±0.023 for Random Forest, ±0.019 for Decision Tree). SVM's gap is larger, +0.011, but it's still comfortably inside its own CV standard deviation (±0.018), so it isn't evidence of anything unusual either. In every case the test score sits within one CV fold's worth of ordinary variation. This ordering tells you almost nothing about overfitting one way or the other. It just means this particular 80/20 test split happened to land on the easy side of average for these four models. A test score meaningfully below the CV mean, or a train/test gap computed directly (Procedure 11.4), is the diagnostic that signals overfitting. A small positive gap like this, even SVM's larger one, is noise, not evidence of a well-generalising model.
Exercise 11.4 [A]
The Worked Example shows Decision Tree accuracy falling from 0.912 (30 features) to 0.789 (2 features), the largest drop of any algorithm. Using this chapter's description of how trees split, explain why restricting to two features hurts a tree more than it hurts Naive Bayes (0.930 → 0.877).
Show the model answer
A Decision Tree builds its predictions entirely out of axis-aligned splits on individual features. Its expressive power comes from chaining many such splits together: each additional feature gives it another axis to cut along, letting it carve out increasingly fine decision regions. Restricted to two features, the tree has nowhere left to turn. It can only ever partition a 2D plane with rectangular cuts, a drastic reduction in what shapes of boundary it can represent. Its accuracy collapses accordingly (0.912 → 0.789). Naive Bayes works completely differently: it multiplies together per-feature likelihoods under an independence assumption, so each feature contributes an essentially separate, additive piece of evidence to the final posterior. Dropping 28 of 30 features just means losing 28 pieces of independent evidence from a sum-like combination. The mechanism itself, multiply what evidence you have, still works exactly the same way with two features as with thirty. It just has less to work with. That's why Naive Bayes degrades gracefully (0.930 → 0.877) while the tree, which fundamentally needs many features to build its structure, degrades sharply.
Exercise 11.5 [E]
A client has a 40,000-row tabular dataset with strongly nonlinear feature interactions and needs predictions to run inside a 5ms latency budget per request. Using Listing 11.1 and this chapter's Trade-offs table, name the algorithm you would rule out first, and explain why using the prediction-time numbers from the Worked Example.
Show the model answer
Rule out K-Nearest Neighbours first. The Trade-offs table already flags KNN as "slow at predict time," and the Worked Example gives the concrete number: 35.4 ms to serve predictions on the held-out test set of only 114 samples drawn from a 455-row training set. That's six times slower than the next-slowest algorithm and over sixty times slower than the fastest. KNN's prediction cost is O(n) in the size of the training set (per the Exposition's KNN section, every prediction requires a distance computation against every stored training point). Scaling from 455 to 40,000 training rows makes that cost far worse, not better. The per-request latency would grow roughly in proportion to the extra data, blowing well past the 5 ms budget, even before accounting for the nonlinear-interaction requirement working against a distance metric already strained by many features. Random Forest or gradient-boosted trees, which handle nonlinear interactions natively and predict in single-digit milliseconds regardless of training set size, are the algorithms worth trying here.
Exercise 11.6 [E]
The No Free Lunch theorem is often summarised as "no algorithm is better than any other." Using the blobs-vs-circles result (1.000 vs. 0.289 for Logistic Regression), explain why that popular summary is misleading for a working data scientist, and state the narrower claim the theorem makes.
Show the model answer
"No algorithm is better than any other" implies a working data scientist should be indifferent between models. That is false and dangerous advice in practice. On the concentric-circles dataset, Logistic Regression scored 0.289, worse than a coin flip, while Random Forest scored a perfect 1.000, on the same data. Two algorithms performed wildly differently on one concrete problem. Nothing about that result says "they're equally good on average across every problem." But it emphatically says they are not equally good on this problem, which is the one the data scientist has to solve. The narrower claim the theorem makes: averaged over all possible problems, including ones with no structure whatsoever, unlike anything encountered in practice, no algorithm has an edge. It says nothing about the much smaller, highly structured set of problems real datasets belong to, where algorithms whose inductive bias matches the data's actual structure, a linear model on linearly-separable blobs, a tree ensemble on nonlinearly-separable circles, reliably win.
Exercise 11.7 [R]
Read Burkov's Chapter 4 on the anatomy of a learning algorithm. Apply its three-part framework (hypothesis representation, loss function, optimisation procedure) to Naive Bayes, and explain what makes its optimisation step so much faster than logistic regression's.
Show the model answer
Discussion only, no single answer. Applying Burkov's framework to Naive Bayes: the hypothesis representation is a set of per-feature conditional probability distributions, one likelihood distribution per feature per class (for GaussianNB, a Gaussian mean and variance for each numeric feature within each class). These are combined multiplicatively via Bayes' theorem to produce a posterior over classes. The loss function is essentially absent in the sense logistic regression uses one. Rather than defining an error to minimise and searching for parameters that reduce it, Naive Bayes' parameters are the maximum-likelihood estimates of each per-feature distribution, computed directly from counts or sample statistics. The optimisation procedure follows from that. The independence assumption reduces parameter estimation to closed-form per-feature statistics (a mean and variance per class, or a frequency count per category). So fitting is a single pass over the training data with no iteration. Contrast Logistic Regression, whose sigmoid-linked linear combination has no closed-form maximum-likelihood solution and requires an iterative convex optimiser (gradient descent or L-BFGS, controlled by max_iter) that repeatedly evaluates and adjusts every coefficient until convergence. That structural difference, closed-form counting versus iterative search, is exactly why Naive Bayes trains in under 2 ms while Logistic Regression takes over 15 ms on the same data, nearly a 10x gap. The gap widens sharply as feature counts grow.