◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Appendix B

Answers to Exercises

Chapter 1 — History & Landscape

Exercise 1.1 [W]. (a) Random Forests: Statistical ML; an ensemble of decision trees learned from labelled examples, no hand-coded rules. (b) BERT: Deep Learning & Foundation Models; a pretrained transformer, part of the post-2012 scale-driven era. (c) MYCIN: Rules & Expert Systems; a hand-coded if-then medical diagnosis system from the 1970s, predating any learning from data.

Exercise 1.2 [W]. They’re nested subsets, not synonyms. Artificial Intelligence is the broadest term: any system exhibiting intelligent behaviour, including hand-coded rule systems. Machine Learning is the subset of AI that learns from data rather than being explicitly programmed. Deep Learning is the subset of ML that specifically uses neural networks. Every deep learning system is machine learning and AI. But not every AI system is machine learning (expert systems aren’t), and not every ML system is deep learning (Random Forests aren’t).

Exercise 1.3 [A]. The answer depends on the three milestones chosen. The mechanical step: append the new rows, then run df_milestones[df_milestones['year'] > 2017] (Procedure 1.2) and count the resulting rows. Any milestone dated 2018 or later (BERT, GPT-3, ChatGPT, GPT-4, etc.) will count; anything from Era 1 or Era 2 will not.

Exercise 1.4 [A]. Reinforcement learning first exceeds classical ML in 2019 (2,231 papers vs. 1,967). That’s four years after deep learning crossed the same line, in 2015. RL’s climb follows the deep reinforcement learning results of the mid-2010s: DeepMind’s DQN playing Atari games (2013, published in Nature in 2015) and AlphaGo (2016). In 2024, reinforcement learning outnumbers classical ML by roughly 1.9x (5,533 vs. 2,850). That’s a much smaller gap than deep learning’s 4.8x over classical ML that year.

Exercise 1.5 [E]. Something like: “With 3,000 labelled examples on a tabular problem, a foundation model is very unlikely to beat a well-tuned XGBoost or LightGBM model on accuracy. It will also cost more to run and be harder to explain to a regulator or auditor. I’d suggest starting with gradient-boosted trees as a baseline this week. If that doesn’t hit the target, we can talk about where a foundation model might help. But it’s not the first thing to reach for here.” The point is to redirect from “AI” as a default answer. The answer should be a specific technique matched to data volume, data type, and interpretability needs.

Exercise 1.6 [E]. Two options. (1) Explicitly filter or mask the first nonzero transition (or any inf/NaN value) before calling nlargest(), e.g. growth.replace([np.inf, -np.inf], np.nan).dropna(). This is the wrong choice if a genuine, very large but finite growth spike should be preserved: a blanket inf filter can be too aggressive if applied without checking what’s being excluded. (2) Add a minimum-volume floor (e.g. only consider percentage growth once the series exceeds some absolute threshold, like 1k papers/year), so early near-zero noise never enters the ranking at all. This is the wrong choice if the point of the analysis is specifically to capture early inflection points before volume is large. That’s exactly what this filter would discard.

Exercise 1.7 [R]. Discussion only. No single answer. Domingos’ essay makes the point (among others) that “more data beats a cleverer algorithm” up to a point, and that feature engineering matters more than the choice of learning algorithm for most practical problems. Applied to Listing 1.7‘s finding: the 2015 crossover and the roughly 5x gap by 2024 could be read as “deep learning is a better algorithm.” Domingos’ framing suggests a different reading. Deep learning unlocked the ability to use far more data, with far less hand-engineered feature work than classical ML. That’s consistent with this chapter’s account of GPU compute and large labelled datasets as the real enablers, not a sudden algorithmic insight in isolation.

Chapter 2 — Business Problem Framing

Exercise 2.1 [W]. “Customer satisfaction” is not measurable or tied to a specific event. It’s a survey construct, not something observable in transactional data at a given moment. A usable label ties it to something concrete and time-bound, e.g. submitted_cancellation_request_within_30d or nps_score_below_6_next_survey.

Exercise 2.2 [W]. (a) Clustering: no target variable, discovering natural groups. (b) Regression: predicting a continuous dollar quantity. (c) Binary classification: fail or not fail within a fixed 48-hour horizon. Predicting which failure mode is a multi-class extension.

Exercise 2.3 [A]. days_since_last_price_increase is safe only if the price increase is known and dated before the prediction point for every row. It can’t be retroactively determined by something that happens after prediction time, such as a price change made in response to early churn signals. If price changes are scheduled independently of individual customer behaviour, the feature is safe. If the business sometimes offers reactive discounts to at-risk customers (which would correlate with the label), it is a leakage risk. It should be excluded, or replaced with a feature computed strictly from data available before the churn-risk window.

Exercise 2.4 [A]. On the class-weighted logistic regression from Listing 2.5, the three thresholds cost $9,345 (0.3: 1,869 false positives, 0 false negatives), $18,255 (0.5: 651 false positives, 75 false negatives), and $24,785 (0.7: 37 false positives, 123 false negatives). 0.3 wins, and by a wide margin. With fp_cost=5 and fn_cost=200, a single missed churner costs as much as 40 false alarms, so the cost-minimising threshold is pulled towards the low end of the swept range even though it means flagging almost every customer in the test set (this model’s probabilities never drop below about 0.32, so 0.3 is close to “flag everyone”). The full 50-point sweep confirms it: the true minimum sits at a threshold of 0.345 ($9,185), just above 0.3, not at some higher, more conservative-looking value.

Exercise 2.5 [E]. A model routing internal support tickets to one of a few roughly equal-sized teams is a reasonable case for accuracy. Misrouting costs a short reassignment step, not a missed high-stakes event. Classes are close to balanced, and the cost of a false positive/negative is symmetric and low. The imbalance argument specifically targets skewed class distributions with asymmetric costs (fraud, churn, disease), where accuracy can be trivially gamed by always predicting the majority class. Symmetric-cost, balanced-class problems don’t have that failure mode.

Exercise 2.6 [E]. With zero historical failure examples, the checklist fails on “Historical data exists (or can be constructed)”: no failure has ever been recorded: supervised anomaly detection isn’t viable yet. The right move is unsupervised/statistical anomaly detection (e.g. an Isolation Forest or simple statistical control limits on sensor readings) to flag unusual readings without labels, paired with a process to have engineers confirm or reject flagged events. That builds the labelled dataset needed for a future supervised model rather than pretending one exists today.

Exercise 2.7 [R]. Discussion only. No single answer. Consider: Wickham’s iterative framing treats “define the problem” as something that is revisited as data exploration reveals what’s measurable. This chapter instead presents the three steps as sequential gates completed before touching data. In practice the two aren’t fully in conflict: the problem statement in Procedure 2.1 is meant to be revised, not treated as immutable. But the chapter’s ordering is a stronger claim than Wickham’s about how much framing must be locked down before EDA begins. Worth considering what breaks if a “final” problem statement is written before any data has been looked at, versus after a first exploratory pass.

Chapter 3 — Maths Foundations: Statistics, Probability & Linear Algebra

Exercise 3.1 [W]. The 99% sensitivity figure describes P(positive∣disease)P(\text{positive} \mid \text{disease}), not P(disease∣positive)P(\text{disease} \mid \text{positive}). Those two conditional probabilities are not the same quantity, despite sounding similar. Bayes’ theorem forces the base rate into the calculation. The disease is rare (1% prevalence), so the pool of healthy people is roughly 99 times larger than the pool of diseased people. Even a modest 5% false-positive rate among that huge healthy majority produces far more false positives in absolute terms than the true positives produced by the small diseased minority testing correctly. The 99% figure describes the test’s behaviour conditioned on already knowing someone is sick. It says nothing about what to conclude before knowing that. That’s exactly the direction the diagnosis question runs.

Exercise 3.2 [W]. (a) Bernoulli: a single binary outcome (spam or not) with no notion of counting across trials. (b) Poisson: counting rare, independent events (fraud) within a fixed interval (an hour), governed by a rate. (c) Binomial: the sum of 50 independent Bernoulli trials (each flip), counting successes. (d) Gaussian: measurement error from many small, independent sources of noise summing together is the textbook Central Limit Theorem scenario.

Exercise 3.3 [A]. Computing df.corr() restricted to sepal width (cm) against the other three iris features gives: petal length (cm) r≈−0.428r \approx -0.428, petal width (cm) r≈−0.366r \approx -0.366, and sepal length (cm) r≈−0.118r \approx -0.118. The weakest (closest to zero) is sepal length (cm) at r≈−0.118r \approx -0.118, negligible by the Correlation Interpretation Guide. This does match the intuition that sepal width is the odd one out. The other three iris measurements are strongly inter-correlated (petal length and petal width alone correlate at r=0.963r=0.963), while sepal width barely tracks any of them, including the other sepal measurement.

Exercise 3.4 [A]. Extending Listing 3.7 to X_raw = df[['petal length (cm)', 'sepal length (cm)']].values and solving gives weights w ~ [-0.0090, 0.4494, -0.0822] (intercept, petal-length coefficient, sepal-length coefficient) and R2≈0.9290R^2 \approx 0.9290. That compares to R2≈0.9271R^2 \approx 0.9271 for the single-feature model. The improvement is real but tiny, about 0.002 of additional variance explained. The new coefficient on sepal length (cm) is small and negative, suggesting it’s picking up a small amount of residual signal (likely correlated with, rather than independent of, petal length) rather than contributing meaningful new information. Adding the second feature does not meaningfully improve the fit.

Exercise 3.5 [A]. Running Procedure 3.4 on petal length (cm) for setosa vs. versicolor gives t≈−39.49t \approx -39.49, p≈5.4×10−62p \approx 5.4 \times 10^{-62}, and Cohen’s d≈−7.90d \approx -7.90. The difference is both statistically significant (the p-value is astronomically below any conventional threshold) and practically enormous. A Cohen’s dd near 8 is far beyond the “large” cutoff of 0.8. These two species’ petal lengths barely overlap at all.

Exercise 3.6 [E]. Before agreeing to ship on “p=0.002p = 0.002” alone: (1) What is the effect size, and is it large enough to matter for the business metric it’s supposed to move? A real but tiny lift can hit p=0.002p=0.002 purely from a large sample size, exactly as the chapter’s 0.9000→0.9001 accuracy example illustrates. (2) What is the confidence interval around the effect, and does its lower bound still represent a meaningful improvement, or does it include effect sizes too small to justify shipping? (3) How many other comparisons were run before this one was reported? Was this the only test, or one of several? If several, an uncorrected p-value overstates the evidence (multiple-comparisons / p-hacking risk).

Exercise 3.7 [E]. Take two features, income (values like 30000, 60000, 90000) and age (values like 25, 45, 65), across three data points: P1=(30000,25)P_1=(30000, 25), P2=(60000,45)P_2=(60000, 45), P3=(90000,65)P_3=(90000, 65). The unscaled Euclidean distance between P1P_1 and P2P_2 is (30000)2+(20)2≈30000.007\sqrt{(30000)^2 + (20)^2} \approx 30000.007. income alone accounts for essentially the entire distance. age‘s contribution is invisible to four significant figures. After standardising both features to mean 0 and standard deviation 1 (via (x - mean) / std, with the population std (ddof=0) across the three points, as StandardScaler uses; pandas’ default .std() uses ddof=1 and gives ±1\pm 1 and a distance of 1.41 instead), each feature’s z-scores follow the same pattern: roughly −1.22,0,1.22-1.22, 0, 1.22 for both, since both are evenly spaced 3-point sequences. The scaled Euclidean distance between P1P_1 and P2P_2 becomes (1.22)2+(1.22)2≈1.73\sqrt{(1.22)^2 + (1.22)^2} \approx 1.73, with income and age now contributing equally. The raw-scale calculation was entirely dominated by the feature with the larger numeric range, even though nothing about the problem says income should matter 1500x more than age.

Exercise 3.8 [R]. Discussion only. No single answer. Candidates worth pursuing: Singular Value Decomposition (A=UΣV⊤A = U\Sigma V^\top) appears only in Appendix A’s Quick Reference. The chapter itself never discusses it. It underlies PCA’s actual numerical implementation: sklearn.decomposition.PCA uses SVD rather than eigendecomposing the covariance matrix directly, for numerical stability. It reappears in Chapter 15’s dimensionality reduction. Alternatively, positive-definiteness and matrix rank: Murphy’s Appendix B covers when X⊤XX^\top X is guaranteed invertible. That connects directly back to this chapter’s warning that the normal equations fail under collinear features, and explains why that failure happens, rather than just stating that it does.

Chapter 4 — Maths Foundations: Calculus & Optimisation

Exercise 4.1 [W]. The gradient’s each component ∂f/∂wi\partial f/\partial w_i tells you how ff changes as wiw_i increases. By construction, moving in the direction of that vector increases ff fastest. That’s what “steepest ascent” means. Since the goal in training is to decrease the loss, the correct move is the exact opposite direction, −∇f(w)-\nabla f(\mathbf{w}). That’s what the gradient descent update rule (w←w−α∇f(w)\mathbf{w} \leftarrow \mathbf{w} - \alpha\nabla f(\mathbf{w})) implements.

Exercise 4.2 [W]. Oscillation followed by a NaN is the “learning rate too high” pattern carried to its extreme. The steps have been overshooting the minimum by a growing margin each time: each overshoot lands further up the opposite wall of the loss surface than the last. Eventually a step produces a value large enough to overflow floating point. The first thing to try is reducing the learning rate by roughly 10x, since Listing 4.2’s table specifically maps both “oscillating” and “NaN or infinity” back to the same root cause.

Exercise 4.3 [A]. At α=1.1\alpha=1.1, the per-step contraction factor is (1−2α)=1−2.2=−1.2(1-2\alpha) = 1 - 2.2 = -1.2, whose magnitude exceeds 1. So instead of shrinking towards zero, xx grows in magnitude every step and flips sign each time. Starting at x=3.0x=3.0, the sequence runs approximately 3.0→−3.6→4.32→−5.184→6.221→−7.4653.0 \to -3.6 \to 4.32 \to -5.184 \to 6.221 \to -7.465 over the first 5 steps, diverging geometrically. This is the 1D version of the same “too large” divergence shown for linear regression in the Worked Example’s Stage 3. The general rule for f(x)=x2f(x)=x^2 is that gradient descent converges only for 0<α<10 < \alpha < 1 (i.e. ∣1−2α∣<1|1-2\alpha|<1).

Exercise 4.4 [A]. At α=0.5\alpha=0.5, five times the original 0.1, the loss reaches the optimum after just 1 epoch: w = [-0.689864, 80.790445], loss 406.5247, an exact match to the closed-form answer, and it stays there for every epoch after. That’s not an approximation rounding down to “epoch 1”; it’s exact. This particular dataset’s Hessian, 2NX⊤X\frac{2}{N}X^\top X, works out to exactly 2I2I: the standardised feature column is, by construction, uncorrelated with the constant bias column, so the two parameters have identical curvature and no cross-term. A learning rate of α=0.5\alpha = 0.5 satisfies α⋅2=1\alpha \cdot 2 = 1 exactly, which zeroes out (I−αH)(I - \alpha H) and collapses gradient descent to the closed-form solution in a single step, regardless of the starting point. This is a property of this specific, well-conditioned, standardised, two-parameter problem, not a general rule for α=0.5\alpha=0.5. It’s a sharper version of the chapter’s “too small wastes epochs” point. On a well-conditioned problem, a larger, still-stable learning rate does not just converge faster. It can hit the exact optimum directly. 0.5 remains well inside the stable range for this loss surface, short of the divergence seen at 1.5.

Exercise 4.5 [A]. Answers vary by the exact value chosen, but on this specific, standardised, two-parameter problem the dynamics are governed exactly by the contraction factor ∣1−2α∣|1-2\alpha| (per Exercise 4.4’s finding that this problem’s Hessian is exactly 2I2I). Any α\alpha between 0 and 1 converges, at different speeds. α=0.3\alpha=0.3 or α=0.7\alpha=0.7 (contraction 0.4) reach within 1% of optimal by epoch 5; α=0.2\alpha=0.2 or α=0.8\alpha=0.8 (contraction 0.6) by epoch 8; α=0.05\alpha=0.05 (contraction 0.9, close to the chapter’s own α=0.1\alpha=0.1) takes about 36 epochs. At exactly α=1.0\alpha=1.0, the contraction factor is ∣1−2∣=1|1-2| = 1: the loss neither grows nor shrinks, oscillating at a fixed distance from the optimum forever rather than converging. Only once α\alpha exceeds 1.0 does the contraction factor exceed 1 and the loss diverge, exactly as α=1.1\alpha=1.1 does in Exercise 4.3. So the real dividing line between “just right” and “too large” on this problem sits at α=1.0\alpha=1.0, not a fuzzy “1.0-1.2” range. The qualitative marker of “too large” is loss increasing rather than decreasing over the run, not merely converging slowly.

Exercise 4.6 [E]. A reasonable experiment: train the same model twice, once with plain SGD and once with Adam. Sweep the initial learning rate over several orders of magnitude (e.g. 10−510^{-5} to 10010^{0}) for each. Plot final loss (or validation accuracy) against the chosen learning rate for both optimisers. The claim predicts Adam’s curve should stay flat and near-optimal across a wide band of learning rates, while SGD’s curve should be sharply peaked: good only in a narrow band, diverging above it and painfully slow below it. The claim would be falsified if Adam’s performance instead degraded about as sharply as SGD’s outside some narrow band, i.e. if the two curves had comparably narrow “good” regions.

Exercise 4.7 [E]. A reasonable response: memory isn’t the problem. L-BFGS-B keeps only the last few gradient differences, about 10 pairs, so at 50 million parameters it needs roughly 20 vectors × 50M × 8 bytes, about 8 GB. That fits on one GPU. The problem is the gradients. Its 2-iteration speed came from exact, full-batch gradients on a smooth convex problem. A neural network is trained on mini-batches, because the training data rarely fits in memory for a full pass, and mini-batch gradients are noisy. That noise breaks L-BFGS-B’s line search and its curvature estimates. For SGD and Adam the same noise helps, because it pushes the optimiser past saddle points on a non-convex loss surface. So the fast convergence you saw doesn’t transfer, and mini-batch SGD or Adam is the right tool here. The square-size Hessian argument applies to exact Newton’s method, which would need about 4,000 GB (3,725 GiB) at only 1 million parameters. It doesn’t apply to L-BFGS.

Exercise 4.8 [R]. Discussion only. No single answer. Goh’s piece frames momentum’s benefit less as “escaping local minima via averaged-out noise” (the SGD story), and more as correcting for ill-conditioning. When a loss surface curves much more steeply in one direction than another (a narrow, elongated valley), plain gradient descent zig-zags across the narrow axis, while making frustratingly slow progress along the long axis. A single learning rate has to be small enough not to diverge on the steep direction, which caps progress on the shallow one. Momentum’s accumulated velocity partially cancels the oscillation across the narrow axis, since opposing updates average out. It reinforces the consistent direction along the long axis, since aligned updates add up. That’s a curvature-shape argument, distinct from and complementary to the “noise helps escape bad minima” story usually told about SGD. Worth noting the two mechanisms address different pathologies (conditioning vs. non-convexity).

Chapter 5 — Information Theory Basics

Exercise 5.1 [W]. Highest to lowest entropy: fair die (H=2.0000H=2.0000 bits) > biased die [0.70,0.10,0.10,0.10][0.70,0.10,0.10,0.10] (H=1.3568H=1.3568 bits) > constant die (H=0.0000H=0.0000 bits). Entropy tracks how spread-out a distribution is. The fair die is maximally unpredictable: all four outcomes are equally likely. The biased die is more predictable, because one outcome dominates. The constant die has zero uncertainty, because the outcome is already known.

Exercise 5.2 [W]. KL(P∥Q)\mathrm{KL}(P\|Q) measures the expected extra cost of using QQ when PP is the true distribution, weighted by how probable each event is under PP. Swapping PP and QQ changes which distribution’s probabilities do the weighting, so the sums aren’t the same. The rule for keeping it straight: the first argument is always “truth” (the actual/reference distribution), the second is always the “approximation” being scored against it.

Exercise 5.3 [A]. Both models classify every example correctly at the 0.5 threshold, but their losses differ because cross-entropy scores confidence calibration, not just which side of the threshold a prediction lands on. y_pred_a = [0.6, 0.6, 0.4, 0.4] is mildly confident about each correct call. Every term is −log⁡(0.6)≈0.5108-\log(0.6) \approx 0.5108 nats, so the average loss is exactly 0.51080.5108. y_pred_b = [0.99, 0.51, 0.49, 0.01] is a mix: two predictions are barely on the correct side (−log⁡(0.51)≈0.6733-\log(0.51) \approx 0.6733 nats each, close to the −log⁡(0.5)≈0.6931-\log(0.5) \approx 0.6931 coin-flip baseline), and two are extremely confident and correct (−log⁡(0.99)≈0.0101-\log(0.99) \approx 0.0101 nats each). Averaging all four gives (0.6733+0.6733+0.0101+0.0101)/4≈0.3417(0.6733 + 0.6733 + 0.0101 + 0.0101)/4 \approx 0.3417 nats: noticeably lower than model A’s 0.51080.5108, not similar or higher. The two barely-confident predictions cost close to the coin-flip baseline each, but the two extremely confident predictions cost almost nothing. Cross-entropy’s cost for a correct call approaches zero as confidence approaches 1, with no floor above zero. That asymmetry means a model with some very confident correct calls can post a lower average loss than a model that plays it uniformly safe, even while carrying a couple of near-toss-up calls. The lesson: 100% classification accuracy at a threshold says nothing about how good the underlying probabilities are. Cross-entropy does, and it doesn’t always penalise a mixed-confidence model more than a uniformly moderate one.

Exercise 5.4 [A]. The entropy works out as

H([0.4,0.4,0.1,0.05,0.05])=−(2×0.4log⁡20.4+0.1log⁡20.1+2×0.05log⁡20.05)≈1.8219 bits.\begin{aligned} H([0.4, 0.4, 0.1, 0.05, 0.05]) &= -\left(2\times 0.4\log_2 0.4 + 0.1\log_2 0.1\right. \\ &\qquad \left. {}+ 2\times 0.05\log_2 0.05\right) \\ &\approx 1.8219 \text{ bits.} \end{aligned}

The five-outcome maximum is log⁡2(5)≈2.3219\log_2(5) \approx 2.3219 bits, so this distribution sits exactly 0.50000.5000 bits below maximum entropy. It’s noticeably more uncertain than the biased four-sided die in the chapter, since probability mass is still split fairly evenly across the top two outcomes.

Exercise 5.5 [A]. For P=[0.6,0.4]P=[0.6,0.4], Q=[0.5,0.5]Q=[0.5,0.5]: KL(P∥Q)=0.6log⁡2(0.6/0.5)+0.4log⁡2(0.4/0.5)≈0.6×0.2630+0.4×(−0.3219)≈0.1578−0.1288=0.0290\mathrm{KL}(P\|Q) = 0.6\log_2(0.6/0.5) + 0.4\log_2(0.4/0.5) \approx 0.6\times0.2630 + 0.4\times(-0.3219) \approx 0.1578 - 0.1288 = 0.0290 bits. KL(Q∥P)=0.5log⁡2(0.5/0.6)+0.5log⁡2(0.5/0.4)≈0.5×(−0.2630)+0.5×0.3219≈−0.13152+0.16096=0.02945≈0.0294\mathrm{KL}(Q\|P) = 0.5\log_2(0.5/0.6) + 0.5\log_2(0.5/0.4) \approx 0.5\times(-0.2630) + 0.5\times 0.3219 \approx -0.13152+0.16096 = 0.02945 \approx 0.0294 bits. That’s close to but not equal to KL(P∥Q)\mathrm{KL}(P\|Q), confirming the asymmetry again. The two happen to be numerically close here only because PP and QQ are close to each other. H(P)=−(0.6log⁡20.6+0.4log⁡20.4)≈0.9710H(P) = -\left(0.6\log_2 0.6 + 0.4\log_2 0.4\right) \approx 0.9710 bits. H(P)+KL(P∥Q)≈0.9710+0.0290=1.0000H(P) + \mathrm{KL}(P\|Q) \approx 0.9710 + 0.0290 = 1.0000 bits. That matches H(P,Q)=−(0.6log⁡20.5+0.4log⁡20.5)=1.0000H(P,Q) = -(0.6\log_2 0.5 + 0.4\log_2 0.5) = 1.0000 bits exactly. Since QQ is uniform over two outcomes, H(P,Q)H(P,Q) reduces to exactly 1 bit regardless of PP. The identity holds.

Exercise 5.6 [E]. A correlation of 0.1 and a mutual-information score of 0.1 are not on the same scale, and don’t mean the same thing. Correlation is bounded in [−1,1][-1,1], with well-understood conventions for “weak.” Mutual information, in nats, as mutual_info_classif returns it, has no fixed scale. Its ceiling is the label’s entropy, and its magnitude depends on the entropy of the specific label and feature distributions in that dataset. A score of 0.1 might be near the noise floor for one problem and a strong, useful signal for another, as the Worked Example showed with feature_noise scoring 0.043 against feature_nonlinear’s 0.338 on the same dataset. The chapter’s Trade-offs section is explicit that MI should only be used to rank features within a single run, not compared against a fixed external threshold. The right move is to always rank features against each other (or against a known-noise baseline feature computed on the same data) rather than picking an absolute cutoff.

Exercise 5.7 [E]. Model answer: a binary label determines which of two shapes a numeric feature is drawn from, both centred at zero. For example, when label is 0, the feature is drawn from a standard normal. When label is 1, the feature is drawn from a mixture that is symmetric around zero but bimodal (e.g., a 50/50 mix of N(−2,0.3)\mathcal{N}(-2, 0.3) and N(2,0.3)\mathcal{N}(2, 0.3)). The mean of the feature is approximately zero regardless of the label, so correlation with the label is near zero. But the shape of the distribution, unimodal versus bimodal, differs sharply between labels. That’s exactly the kind of higher-order dependence mutual information, but not correlation, is sensitive to. This is a generalisation of the Worked Example’s variance-based trick to an even more clearly non-linear signal (shape rather than spread).

Exercise 5.8 [R]. Discussion only. No single answer. Shannon’s paper introduces channel capacity and the noisy-channel coding theorem, which this chapter never touches. The idea: a communication channel has a maximum rate at which information can be transmitted with arbitrarily low error, and entropy sets the fundamental limit on lossless compression. Worth considering how this connects to the chapter’s ML framing: a classifier’s cross-entropy loss floor at H(P)H(P) is structurally the same idea as a channel’s capacity limit. No amount of cleverness can push performance past a limit set by the intrinsic uncertainty of the source. Students should also notice that Shannon’s original motivation was communication engineering, not statistics or machine learning, which came decades later and inherited the same mathematics.

Chapter 6 — Python Data Science Ecosystem

Exercise 6.1 [W]. pandas DataFrames store each column as a NumPy array under the hood, so pandas’ speed and much of its behaviour (vectorised operations, dtype handling) come directly from NumPy. But NumPy arrays have no concept of labelled rows, mixed column types, or an index, so NumPy has no need to understand pandas at all. The dependency runs one way, matching the layered-stack diagram in the Exposition.

Exercise 6.2 [W]. .fit() learns parameters from training data. .transform() applies a learned transformation to produce new data, used for preprocessing. .predict() applies a learned supervised model to produce predictions on new data. .transform() and .predict() are safe to call on test data. .fit() should never be called on test data, since fitting on it leaks test-set information into the transformation or model.

Exercise 6.3 [A]. Timing np.sqrt against an equivalent [math.sqrt(x) for x in py_list] loop over 500,000 elements gives roughly 22.5 ms for the Python loop and roughly 0.46 ms for the NumPy call: a speedup of approximately 49x. This is smaller than Listing 6.6’s ~117x multiplication speedup, because math.sqrt is a fast built-in per call, narrowing (but not closing) the gap. The qualitative result, NumPy wins by well over an order of magnitude, holds regardless of which elementwise operation is timed.

Exercise 6.4 [A]. Pivoting survival rate by pclass and embark_town on the Titanic dataset shows the lowest survival rate is third class passengers who boarded at Southampton, at roughly 19%, noticeably lower than third-class Cherbourg (38%) or Queenstown (38%) passengers. This is a case where a single grouping variable (class alone, or port alone) would have hidden the compounding effect visible only in the two-dimensional pivot.

Exercise 6.5 [A]. Comparing age between survivors (mean ≈\approx 28.3 years) and non-survivors (mean ≈\approx 30.6 years) with scipy.stats.ttest_ind gives t ~ -2.07, p ~ 0.039. That’s below the 0.05 threshold, so the age difference is statistically significant, with survivors being slightly younger on average. The effect is much weaker than the fare difference in Listing 6.8 (p ≈\approx 6e-15). Age separates survivors from non-survivors only modestly, while fare (as a proxy for class) separates them dramatically.

Exercise 6.6 [E]. If a scaler were fit on test data, or on the combined train+test set, rather than only on training data, the reported test accuracy would be inflated. The scaler’s mean and standard deviation would already “know” something about the distribution of the test set. That gives the model an information advantage it wouldn’t have in production, where future data is by definition unseen at fit time. No exception is raised because nothing about calling .fit() on the wrong slice of data is syntactically or semantically invalid. scikit-learn has no way to know which rows are supposed to represent “the future.” So the leak is silent, and only shows up later as a mismatch between validation performance and real-world performance.

Exercise 6.7 [E]. Twenty columns of mixed numeric and categorical data is close to a worst case for raw NumPy. NumPy arrays are single-typed, so mixed columns would need to be split across multiple arrays, or coerced into a single dtype, usually object, which forfeits NumPy’s speed advantage entirely. Every join, filter, or groupby that pandas expresses in one readable line would need to be hand-written with manual index bookkeeping. The “for speed” framing is backwards here: pandas’ groupby and filtering operations already delegate to vectorised NumPy underneath. The actual speed cost of raw NumPy on this kind of data would be development time, not runtime.

Exercise 6.8 [R]. Discussion only, no single answer. Candidates worth investigating in the NumPy quickstart: the rule that dimensions are compared from the trailing (rightmost) axis, and either must match or one of them must be 1 or absent. Or that broadcasting can silently succeed on shapes the author didn’t intend to be compatible: e.g. a (3,) array broadcasting against a (3, 1) array to produce a (3, 3) result via outer-product-like expansion, rather than the elementwise operation the author expected. Consider a concrete case: adding a “per-row mean” array of shape (n,) back onto a (n, m) matrix without reshaping it to (n, 1) first. Broadcasting rules make this fail loudly in some shapes and silently produce the wrong answer in others, depending on whether n happens to equal m.

Chapter 7 — Data Cleaning & Wrangling

Exercise 7.1 [W]. (a) MCAR: a hardware fault dropping readings at random has no relationship to the reading’s value or any other variable. (b) MNAR: the probability of a missing income value depends directly on the (unobserved) income itself, higher earners are more likely to decline. (c) MAR: missingness correlates with an observed variable (signup channel), not with the true age value itself; conditional on signup channel, the missing ages aren’t systematically different from the reported ones.

Exercise 7.2 [W]. The typo assignment ages[0] = 312 is applied after the missing-value assignment, so that it deterministically overwrites row 0, regardless of whether index 0 happened to be selected by np.random.choice(n, 40, replace=False). If the two lines were swapped, row 0 would first be set to 312. Then, only if index 0 were among the 40 randomly chosen missing indices, it would get overwritten back to NaN. That would make the presence of the planted outlier depend on random chance, rather than being guaranteed. The chapter’s ordering guarantees the outlier exists every time the notebook runs with a fixed seed.

Exercise 7.3 [A]. The one-line change is df['age'] = df['age'].fillna(df['age'].mean()) in place of .median(). After deduplication and before capping, age has a mean of 42.82 against a median of 42, the 312 outlier pulling the mean up above the rank-based median. Capping that single value down to 90 drags the mean back down to about 42.34, very close to the median of 42. So mean imputation would land at roughly 42.3, a third of a point above the reported median-imputed value of 42. The small remaining gap comes from the mild right skew still left in age even after the domain cap.

Exercise 7.4 [A]. Q1 and Q3 are the 25th and 75th percentile of the whole column. Only one value, 312, sits far above the bulk of the data (ages spread across 22–64). That single point has almost no leverage on the rank-based Q1/Q3 statistics. 312 is one point out of hundreds, so the 25th/75th percentile boundaries barely move to accommodate it. The resulting IQR fence lands well below 312, meaning IQR does in fact catch it. That’s the reported “1 outlier.” The subtler risk is the opposite one: IQR is a whole-column, single-pass method. A handful of outliers or a long natural tail can each be caught or missed, depending on how many other values sit between the bulk of the distribution and the fence. That’s why the chapter still applies a domain-knowledge cap ([16, 90]) on top of IQR, rather than trusting IQR’s fence alone.

Exercise 7.5 [E]. A pipeline running without errors only confirms that every operation was syntactically and type-valid. It says nothing about whether the result is semantically correct. A concrete silent failure: an incorrectly specified gender_map that’s missing the 'MALE' (all-caps) variant wouldn’t raise any error. .map() would convert every 'MALE' entry to NaN, silently turning a known category into missing data. The pipeline would report success right up until someone later asks why 50 rows have a missing gender that used to have a value.

Exercise 7.6 [E]. Sample rules: (1) (d['order_total'] > 0).all(), no order should have a non-positive total; (2) (d['discount_pct'] >= 0).all() and (d['discount_pct'] <= 100).all(), a discount percentage must be a valid percentage; (3) d['customer_id'].notnull().all(), every order must be attributable to a customer; (4) a cross-column rule: (d['delivery_date'] >= d['ship_date']).all(), a package cannot be delivered before it ships. Single-column rules can never catch that, since both dates can individually look perfectly valid. A fifth candidate, if the data also had an order_date column: (d['ship_date'] - d['order_date']).dt.days.between(0, 30).all() as a sanity bound on fulfilment time, catching data-entry errors that swap order and ship dates.

Exercise 7.7 [R]. Discussion only, no single answer. Multiple imputation with a missingness indicator is the most tractable technique to start with. Alongside the imputed value, an explicit binary column recording whether the value was originally missing is added as a feature. That lets a downstream model learn that “this value was estimated” is itself informative, relevant when missingness correlates with the outcome, as in MNAR. A Heckman correction goes further, formally modelling a “selection equation” for whether a value is observed at all, jointly with the equation of interest. It requires an instrument variable that affects observability but not the outcome directly. That’s often the hardest part to find in practice. Worth considering: unlike MCAR/MAR imputation, none of these techniques make the MNAR problem disappear. They make its assumptions explicit and testable rather than silently violated.

Chapter 8 — SQL for Data Science

Exercise 8.1 [W]. Standard SQL evaluates HAVING before column aliases defined in the SELECT list are resolved. The aggregate expression SUM(revenue) exists as soon as the grouping happens, but the alias total_revenue is only attached to the output column afterwards, so HAVING can’t yet “see” it. PostgreSQL, SQL Server and Oracle follow the standard and reject the alias. DuckDB, MySQL and SQLite accept it as an extension. Repeating the full aggregate expression (HAVING SUM(revenue) > 1000) works because it doesn’t depend on the alias having been assigned.

Exercise 8.2 [W]. With three rows tied for first place: ROW_NUMBER() breaks the tie arbitrarily (by whatever order the engine processes ties in) and assigns 1, 2, 3. Every row gets a distinct number even though they’re tied on the ordering column. RANK() assigns all three rows 1, 1, 1 and then the next row after them jumps to 4, a gap appears that skips one fewer than the number of tied rows. DENSE_RANK() also assigns all three rows 1, 1, 1, but the next row gets 2, no gap. The choice matters operationally. ROW_NUMBER() is right for “give me exactly N rows”: e.g. top 3, no ties allowed. RANK() is right when ties should be visible as ties, with the gap reflecting how many rows share that rank. DENSE_RANK() is right for “how many distinct rank levels are there”: e.g. medal tables, ignoring how many people share a level.

Exercise 8.3 [A]. Adapting Listing 8.12 to GROUP BY c.region (joining orders to customers instead of products) gives, on the seed-42 dataset: East leads with $564,538.19 across 715 orders, West is a very close second at $564,314.41 across 666 orders, a difference of only $223.78. That’s well within noise for a synthetic dataset of this size, and effectively a tie. North ($502,071.81, 557 orders) and South ($486,219.92, 591 orders) trail more noticeably. Unlike the category breakdown, where Sports led Clothing by a clear $76,509.76 margin, the regional split shows no real leader. East and West are statistically indistinguishable here, a reminder that “which group is #1” can be a fragile question when the top two are this close.

Exercise 8.4 [A]. Changing the retention window from 90 to 60 days in Listing 8.14 (same Q1 2022 cohort of 156 customers) drops the retained count from 128 (82.1%, the chapter’s reported 90-day figure) to 99 (63.5%). That’s an 18.6 percentage point difference. That sensitivity to the window length is evidence for the chapter’s caution about the original 90-day figure. A retention rate that swings by nearly 19 points when the window shrinks by a third is a symptom of orders scattered close to uniformly across the observation period. As the Worked Example explains, orders were assigned to customers uniformly at random over 730 days. It isn’t reflecting genuine, sustained re-engagement behaviour. A retention metric this sensitive to an arbitrary window choice needs the underlying data-generating assumptions checked before it’s trusted at face value.

Exercise 8.5 [A]. Re-running Listing 8.13 over the full (unlimited) customer ranking shows the cumulative percentage crosses 50% at the 74th customer (49.68% at rank 73, 50.21% at rank 74). So 74 of the 200 customers, or 37.0% of the customer base, are needed to reach half of total completed revenue. That’s close to, but slightly past, the “half the revenue from roughly a third of customers” pattern the chapter’s Worked Example already flags as flatter than a genuine 80/20 Pareto split would produce.

Exercise 8.6 [E]. One concrete change: instead of drawing quantity uniformly from randint(1, 6) and price uniformly from uniform(5, 500), draw per-customer purchase frequency from a power-law / Zipf distribution, e.g. np.random.zipf(a=2, size=n_customers), to set how many orders each customer places. Combine that with a log-normal price distribution, np.random.lognormal(mean, sigma) instead of uniform prices, for products. Both changes matter for the same underlying reason. A Pareto (80/20) pattern emerges specifically from multiplicative, heavy-tailed processes: a small number of customers who both order far more frequently and skew towards higher-priced products. Independent uniform draws for quantity, price, and order assignment guarantee every customer’s expected revenue converges towards the same value as sample size grows. That mechanically flattens the concentration curve, regardless of how many orders are simulated.

Exercise 8.7 [E].

SELECT
    category,
    SUM(CASE WHEN status = 'completed'
             THEN revenue ELSE 0 END) AS completed_revenue,
    SUM(CASE WHEN status = 'returned'
             THEN revenue ELSE 0 END) AS returned_revenue
FROM orders o
JOIN products p ON o.product_id = p.product_id
GROUP BY category;

This is preferable to two separate queries joined afterwards. It scans the orders/products join exactly once instead of twice. The database only needs to plan and execute a single query. The result arrives as one row per category, with both figures already aligned. No separate join step is needed in pandas afterwards to line the two result sets up by category. It also guarantees consistency: both numbers are computed from the exact same snapshot of the underlying tables. Two independently-run queries executed moments apart are not strictly guaranteed to be, if the underlying data is being concurrently modified.

Exercise 8.8 [R]. Discussion only, no single answer. The logical evaluation order (FROM → WHERE → GROUP BY → HAVING → window functions → SELECT → ORDER BY) places window functions after WHERE and HAVING. By then, the row set is down to whatever rows and groups survive filtering. Window functions still run before the final SELECT list decides which computed columns to display. That ordering is why a window function can’t appear inside a WHERE clause directly. At the point WHERE is evaluated, no window function has been computed yet. There’s nothing for the filter to reference. It’s also why filtering on a window function’s result requires wrapping the whole query in a CTE or subquery: compute the window function inside, then filter with an outer WHERE. That’s exactly the shape Listing 8.8 and procedures like Procedure 8.5 and the “top-N per group” skeleton in this chapter’s Quick Reference use. Worth connecting this to why HAVING (Exercise 8.1) has the same “can’t reference something not yet computed” character: both restrictions come from the same clause-ordering rule, just applied to aggregates versus window functions respectively.

Chapter 9 — Exploratory Data Analysis & Visualisation

Exercise 9.1 [W]. The five questions, in order: (1) what do I have (shape, types, nulls, duplicates); (2) what does each variable look like alone (distributions, outliers, skew); (3) how do variables relate to each other (correlations, conditional distributions); (4) how do variables relate to the target; (5) what surprises me. “What surprises me?” is placed last because it can only be answered after the first four questions have established a baseline of expectations. You can’t recognise a surprise until you know what “normal” looks like for this specific dataset.

Exercise 9.2 [W]. A mean of 29.7 against a median of 28.0 means the mean is pulled upward relative to the median. That indicates a right-skewed (positive skew) distribution: a small number of unusually old passengers/customers/etc. stretching the upper tail. A histogram with a KDE overlay (Listing 9.4) would confirm the shape directly and show whether the skew is mild or extreme.

Exercise 9.3 [A]. Answer depends on the dataset chosen. The deliverable is a quick_profile() run and a reported list of columns exceeding 10% missing, following the exact output format of Listing 9.2. For example, on the Titanic dataset used in this chapter, only deck (77.2%) and age (19.9%) exceed that threshold.

Exercise 9.4 [A]. df.groupby('embarked')['survived'].mean() on the Titanic dataset returns roughly 55% survival for passengers who embarked at Cherbourg (C), 39% at Queenstown (Q), and 34% at Southampton (S). The spread (~20 points between C and S) is large enough to be worth investigating as a feature. It is confounded with class, though: Cherbourg passengers were disproportionately 1st class, so some of this apparent effect is the class/fare effect from Listing 9.8 showing through a correlated variable.

Exercise 9.5 [A]. Running Listing 9.5’s near-redundant-pair check on survived, pclass, age, sibsp, parch, fare finds no pair above the 0.95 threshold. The strongest pairwise correlation among the features (excluding the target) is pclass/fare at roughly −0.55-0.55, well below the redundancy cutoff. This dataset has essentially no redundant numeric feature pairs, so no columns need to be dropped purely on multicollinearity grounds.

Exercise 9.6 [E]. A plausible candidate is fare split by pclass and survived jointly. The marginal correlation between fare and survived (0.26) is real, but could hide a stronger within-class effect, since fare varies enormously within 1st class, from a low-end 1st-class ticket to the most expensive suites. A passenger paying a top fare within 1st class might show a bigger survival boost than the marginal correlation suggests. The general pattern to watch for: any feature whose marginal relationship to the target looks weak but which is known to vary systematically with a third, stronger predictor (here, class) is a candidate for the same hidden effect.

Exercise 9.7 [E]. Running EDA on the full dataset, test rows included, contaminates every pattern it finds. An outlier threshold, a “this feature looks predictive” judgement, or a bucketing scheme for rare categories has then been chosen with knowledge of the test set’s actual values. Even without directly touching the labels, decisions like “drop rows where fare > 400” or “bucket embarked’s rare category into ‘other’” become choices tuned to the specific data points that will later be used to evaluate the model. That inflates the model’s apparent generalisation performance. The fix is to split first, then run full EDA only on the training partition, treating the test set as if it doesn’t exist until final evaluation.

Exercise 9.8 [R]. Discussion only, no single answer. Each of Anscombe’s four datasets shares an identical mean, variance, and Pearson correlation coefficient (~0.816). Summary statistics alone would report the same numbers for all four, and suggest the same linear relationship strength. What they hide: one dataset is a genuine linear relationship with noise. One is a clean non-linear (quadratic-like) curve that Pearson’s linear correlation badly misrepresents. One is a near-perfect linear relationship broken by a single outlier that shifts the fitted line. And one is a vertical cluster of points entirely determined by one high-leverage outlier. Only plotting each one reveals which of these four very different stories the identical statistics were describing.

Chapter 10 — The Data Science Lifecycle

Exercise 10.1 [W]. 1. Problem framing: Chapter 2. 2. Data collection & understanding: Chapters 7-9. 3. Data preparation: Chapters 7 and 14. 4. Modelling: Chapters 11-20. 5. Evaluation: Chapters 13 and 16. 6. Deployment: Book 2, Chapter 23. 7. Monitoring: Book 2, Chapter 25. 8. Iteration: loops back to Chapter 2’s problem framing.

Exercise 10.2 [W]. The left-to-right pipeline diagram implies each stage is finished once and handed off to the next, but real projects loop backward constantly: evaluation (stage 5) routinely reveals the success metric itself was wrong. A model hits a strong AUC, the team celebrates, and the business owner then explains they cared about precision at a specific operating point nobody optimised for, sending the project back to stage 1. Similarly, modelling (stage 4) routinely sends a project back to data preparation (stage 3) when a model’s errors reveal a feature bug or missing signal. The two stages cycle many times rather than completing in sequence. A diagram with a single feedback arrow tacked on at the end can’t represent a process where nearly every stage has an edge back to nearly every earlier one.

Exercise 10.3 [A]. Take this ask dictionary:

ask = {
    "simple_rule_suffices": False,
    "one_off_analysis_only": False,
    "historical_labelled_data_exists": False,
}

It describes, say, a stakeholder who wants to predict equipment failures but has never logged which equipment failed. It returns "Not ready -- this is a data collection project first, an ML project second". This is the correct outcome because Listing 10.3 checks simple_rule_suffices first (false here, so it doesn’t short-circuit to “ship a rule”), then whether this is a one-off analysis (it isn’t), then whether historical labelled data exists. It doesn’t, so no amount of modelling skill can produce a supervised model yet. The correct next step is building a labelling/data-collection process, not starting a lifecycle project that has no label to train against.

Exercise 10.4 [A]. On a dataset with a 3% positive rate, the naive “predict majority class” baseline (always predict the negative class) scores 97% accuracy by construction. It is the negative class’s share of the data. That number is a poor argument for production use because the baseline achieves it while catching zero positives. Precision and recall on the positive class are both 0.00. The model is useless for any business action that depends on identifying positives. This is the exact scenario the tutorial’s dummy classifier shows at 93% accuracy and 7% churn. A high accuracy figure on an imbalanced dataset says nothing about whether the model does the one thing it’s being built for.

Exercise 10.5 [A]. The two stages to loop back to are stage 3 (data preparation) and stage 1 (problem framing/success metric). For stage 3: add richer features. The tutorial notebook works from only six raw features: support tickets, payment failures, login recency, tenure, plan type, product count. Engineering additional signals, such as usage trend over time, interaction-weighted engagement scores, or billing-cycle-relative features (Chapter 14), could give the model more to separate churners from non-churners than a modest AUC of 0.638 suggests is currently available. For stage 1: renegotiate the success metric. Precision ≥\geq 0.40 at recall ≥\geq 0.60 may be unattainable with the signal currently available. The business owner may need to accept a different operating point, such as lower recall in exchange for usable precision. Or they may need a different intervention design, such as a broader, cheaper retention outreach that tolerates lower precision. The alternative is asking the model to hit a target the data can’t support yet.

Exercise 10.6 [E]. Moving from nightly batch scoring to real-time scoring at the moment of a support contact changes all three elements the tutorial’s Stage 6 plan assumed. The serving pattern shifts from a scheduled batch job over the full customer base to a request-response endpoint invoked synchronously (or near-synchronously) when a support interaction begins. The model needs to be behind a low-latency API, e.g. a SageMaker real-time endpoint, rather than a nightly Batch Transform job. Features need to be computable on demand rather than pre-joined the night before, which likely requires a feature store with fresh, queryable feature values, rather than a batch feature table. The latency budget tightens dramatically. Batch scoring can tolerate minutes to hours per run. A support-interaction-triggered prediction needs to return in well under a second, so it doesn’t stall the support agent’s workflow. This rules out any model or feature pipeline with expensive real-time joins or slow feature computation. The rollback plan also changes. A bad batch run can be discarded and rerun before anyone sees the output. A bad real-time model is affecting live support decisions immediately. So the rollback plan needs an instant traffic-shifting mechanism, e.g. an endpoint alias or canary rollback, rather than “rerun the job.” It also needs a fallback response, such as defaulting to the old rule-based flag, if the real-time endpoint is unavailable or too slow.

Exercise 10.7 [E]. CRISP-DM under-specifies monitoring. Per Appendix A’s Framework Comparison, it has no explicit monitoring phase at all, ending effectively at deployment. A team that “follows CRISP-DM” literally has no process step that requires anyone to watch the model after it ships. No dashboards, no alerts, and critically no named owner for production health. That explains exactly the pattern described: models get shipped and never revisited. TDSP would have required an explicit deployment-as-ongoing / customer-acceptance stage with monitoring dashboards, drift alerts, and, per Appendix A’s RACI table, a named individual (not “the team”) accountable for receiving those alerts and triggering retraining. The gap isn’t a modelling failure. It’s a process framework that never asked “who watches this after ship day.”

Exercise 10.8 [R]. Discussion only, no single answer. Worth examining directly: the original CRISP-DM paper treats deployment as its sixth and final phase, with only a brief note that the process is “cyclic.” It offers no dedicated deliverable, checklist, or owner for what an iteration requires. Iteration is a property asserted of the diagram, not a phase with its own exit criteria. TDSP’s documentation, by contrast, explicitly draws the lifecycle looping back into Business Understanding, and ties that loop to concrete artefacts: updated project charters, retrained model versions. The difference suggests CRISP-DM, written in 1996 for data-mining projects that often ended at a delivered analysis or report, didn’t anticipate that “the model runs continuously in production and needs to be revisited” would become the normal case. TDSP, written in the cloud/MLOps era, treats the loop-back as the expected steady state of a live system, not an edge case.

Chapter 11 — Supervised Learning: Overview of Approaches

Exercise 11.1 [W]. (a) Spam detection using word-count features: Probabilistic (Naive Bayes). Its core assumption is that features are conditionally independent given the class, which is a reasonable approximation for bag-of-words counts given a topic. (b) Predicting house price from square footage and location: Linear. Its core assumption is that the target is well-approximated by a linear combination of the input features. (c) Classifying flowers from petal measurements using their five closest labelled neighbours: Distance-Based (KNN). Its core assumption is that nearby points in feature space share a label, so a new point’s class is decided by a majority vote among its closest training examples.

Exercise 11.2 [W]. The equation is Expected Test Error=Bias2+Variance+Irreducible Error\text{Expected Test Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}. A shallow tree can only express a small number of coarse if/else rules, so it’s forced to represent the true decision boundary poorly no matter what training data it sees. That’s high, stable bias. Letting the tree grow deeper adds more splits, so it can carve out increasingly fine-grained regions that match the true boundary more closely, which is exactly what lowers the bias term. But those extra splits are fit to the specific noise in the training sample, and a different training set would produce different deep splits, so the fitted function becomes unstable across resamples, which is exactly what raises the variance term. Flexibility is a dial on how much the model is allowed to bend to fit what it’s shown. Bending further to reduce systematic error is the same mechanism that makes the fit more sensitive to which particular examples it saw.

Exercise 11.3 [A]. Four algorithms in the benchmark table show test accuracy exceeding CV mean accuracy: Logistic Regression (0.982 vs. 0.980, +0.002), SVM (0.982 vs. 0.971, +0.011), Random Forest (0.956 vs. 0.954, +0.002), and Decision Tree (0.912 vs. 0.910, +0.002). Three of those four gaps are tiny and identical, +0.002, negligible next to the CV standard deviations reported alongside them (±0.013 for Logistic Regression, ±0.023 for Random Forest, ±0.019 for Decision Tree). SVM’s gap is larger, +0.011, but it’s still comfortably inside its own CV standard deviation (±0.018), so it isn’t evidence of anything unusual either. In every case the test score sits within one CV fold’s worth of ordinary variation. This ordering tells you almost nothing about overfitting one way or the other. It just means this particular 80/20 test split happened to land on the easy side of average for these four models. A test score meaningfully below the CV mean, or a train/test gap computed directly (Procedure 11.4), is the diagnostic that signals overfitting. A small positive gap like this, even SVM’s larger one, is noise, not evidence of a well-generalising model.

Exercise 11.4 [A]. A Decision Tree builds its predictions entirely out of axis-aligned splits on individual features. Its expressive power comes from chaining many such splits together: each additional feature gives it another axis to cut along, letting it carve out increasingly fine decision regions. Restricted to two features, the tree has nowhere left to turn. It can only ever partition a 2D plane with rectangular cuts, a drastic reduction in what shapes of boundary it can represent. Its accuracy collapses accordingly (0.912 → 0.789). Naive Bayes works completely differently: it multiplies together per-feature likelihoods under an independence assumption, so each feature contributes an essentially separate, additive piece of evidence to the final posterior. Dropping 28 of 30 features just means losing 28 pieces of independent evidence from a sum-like combination. The mechanism itself, multiply what evidence you have, still works exactly the same way with two features as with thirty. It just has less to work with. That’s why Naive Bayes degrades gracefully (0.930 → 0.877) while the tree, which fundamentally needs many features to build its structure, degrades sharply.

Exercise 11.5 [E]. Rule out K-Nearest Neighbours first. The Trade-offs table already flags KNN as “slow at predict time,” and the Worked Example gives the concrete number: 35.4 ms to serve predictions on the held-out test set of only 114 samples drawn from a 455-row training set. That’s six times slower than the next-slowest algorithm and over sixty times slower than the fastest. KNN’s prediction cost is O(n)O(n) in the size of the training set (per the Exposition’s KNN section, every prediction requires a distance computation against every stored training point). Scaling from 455 to 40,000 training rows makes that cost far worse, not better. The per-request latency would grow roughly in proportion to the extra data, blowing well past the 5 ms budget, even before accounting for the nonlinear-interaction requirement working against a distance metric already strained by many features. Random Forest or gradient-boosted trees, which handle nonlinear interactions natively and predict in single-digit milliseconds regardless of training set size, are the algorithms worth trying here.

Exercise 11.6 [E]. “No algorithm is better than any other” implies a working data scientist should be indifferent between models. That is false and dangerous advice in practice. On the concentric-circles dataset, Logistic Regression scored 0.289, worse than a coin flip, while Random Forest scored a perfect 1.000, on the same data. Two algorithms performed wildly differently on one concrete problem. Nothing about that result says “they’re equally good on average across every problem.” But it emphatically says they are not equally good on this problem, which is the one the data scientist has to solve. The narrower claim the theorem makes: averaged over all possible problems, including ones with no structure whatsoever, unlike anything encountered in practice, no algorithm has an edge. It says nothing about the much smaller, highly structured set of problems real datasets belong to, where algorithms whose inductive bias matches the data’s actual structure, a linear model on linearly-separable blobs, a tree ensemble on nonlinearly-separable circles, reliably win.

Exercise 11.7 [R]. Discussion only, no single answer. Applying Burkov’s framework to Naive Bayes: the hypothesis representation is a set of per-feature conditional probability distributions, one likelihood distribution per feature per class (for GaussianNB, a Gaussian mean and variance for each numeric feature within each class). These are combined multiplicatively via Bayes’ theorem to produce a posterior over classes. The loss function is essentially absent in the sense logistic regression uses one. Rather than defining an error to minimise and searching for parameters that reduce it, Naive Bayes’ parameters are the maximum-likelihood estimates of each per-feature distribution, computed directly from counts or sample statistics. The optimisation procedure follows from that. The independence assumption reduces parameter estimation to closed-form per-feature statistics (a mean and variance per class, or a frequency count per category). So fitting is a single pass over the training data with no iteration. Contrast Logistic Regression, whose sigmoid-linked linear combination has no closed-form maximum-likelihood solution and requires an iterative convex optimiser (gradient descent or L-BFGS, controlled by max_iter) that repeatedly evaluates and adjusts every coefficient until convergence. That structural difference, closed-form counting versus iterative search, is exactly why Naive Bayes trains in under 2 ms while Logistic Regression takes over 15 ms on the same data, nearly a 10x gap. The gap widens sharply as feature counts grow.

Chapter 12 — Ensemble Methods

Exercise 12.1 [W]. If BB models each have variance σ2\sigma^2 and their errors are uncorrelated, averaging them gives variance σ2/B\sigma^2/B, strictly smaller as BB grows, with no term in that formula affecting bias. Bagging doesn’t change what each individual model is trying to predict: no change to the systematic error, i.e. bias. It only reduces how much the ensemble’s average prediction fluctuates around that target across different training samples.

Exercise 12.2 [W]. (a) Boosting: a model systematically wrong on a subgroup has high bias on that subgroup, which boosting directly targets by reweighting/refitting on exactly the hard cases. (b) Bagging: wildly changing predictions from small training-data changes is the definition of high variance, which bagging’s averaging over bootstrap samples is designed to reduce.

Exercise 12.3 [A]. Random Forest’s RMSE barely moving from 200 to 500 trees ($45,502 → $45,514, marginally worse, well within noise) indicates bagging’s variance reduction has already saturated before 200 trees: additional trees add averaging stability with diminishing, essentially flat returns. LightGBM keeps improving slightly over the same range ($28,323 → $28,159), because boosting is still reducing bias round by round at 200 trees. Each new tree is still correcting real signal the ensemble hasn’t captured yet, not just fitting noise. That’s the core asymmetry this chapter’s Exposition draws between the two. Bagging’s averaging effect plateaus once the ensemble is large enough to cancel variance. Boosting keeps extracting value from additional rounds for longer, up to the point, not yet reached here, where it starts overfitting instead.

Exercise 12.4 [A]. LightGBM, not XGBoost, is the single best individual model in this chapter’s benchmark: $28,323 RMSE, versus XGBoost’s $28,873. It’s already one of the two models in the existing stack. That makes beating solo XGBoost’s $28,873 specifically a fairly low bar: LightGBM alone already clears it, so the three-model stack would need the meta-learner to badly mis-weight things for the combined prediction to land worse than $28,873. The more interesting, harder bar is solo LightGBM’s own $28,323. This chapter’s own measured result is a reason for caution there: the RF+LightGBM stack’s meta-learner assigned Random Forest, the weaker base model, a negative weight, and the stack still ended up worse than solo LightGBM ($29,535 vs. $28,323). Adding XGBoost, weaker than LightGBM but stronger than Random Forest, could get down-weighted the same way and land close to solo LightGBM, or dilute it enough to land worse. Expect the three-model stack to beat solo XGBoost comfortably. Don’t assume it beats solo LightGBM without checking.

Exercise 12.5 [E]. “Which models were in the stack, and did you compare against the single best individual model before concluding the stack is better? This chapter’s own measured result shows a stack can be worse than the best model it was built from, even with that model inside the stack. ‘Stacked and improved over all five individually’ needs to be checked against the best of the five, not just an average or the worst of them.”

Exercise 12.6 [E]. Random Forest-first is right when time-to-baseline matters more than squeezing out the last accuracy, since it needs almost no tuning and gives a solid, honest number quickly (per Procedure 12.3). It’s the wrong default when the roughly 58% RMSE gap this chapter measured (Random Forest $45,502 vs. XGBoost $28,873) matters to the business outcome. At that point, the extra tuning effort XGBoost/LightGBM demand is worth paying for, and “simpler” shouldn’t be conflated with “good enough.”

Exercise 12.7 [R]. Discussion only. Likely candidates include XGBoost’s cache-aware block structure for out-of-core computation (enabling training on datasets larger than memory) or its weighted quantile sketch algorithm for approximate split-finding on weighted data. Either addresses a scalability problem (very large datasets, or weighted training samples) beyond what this chapter’s Trade-offs table, which focuses on bias/variance/overfitting risk, covers.

Chapter 13 — Cross-Validation & Model Selection

Exercise 13.1 [W]. Choosing hyperparameters based on test-set performance means the model was implicitly optimised, at least in part, to do well on that exact set of examples. The test set has stopped being purely “unseen data.” It has become part of the selection process. Any resulting test-set score is now measuring how well the model fits that specific set, not how well it generalises to new data.

Exercise 13.2 [W]. (a) Leave-One-Out: 30 patient records is well under the “<50 examples” threshold for LOO. (b) TimeSeriesSplit: daily sales are strictly time-ordered, so random shuffling would leak the future into training. (c) Stratified k-fold (k=5): a roughly balanced, moderate-size dataset is the standard case Stratified k-fold with k=5 is designed for.

Exercise 13.3 [A]. The ten seed-accuracy values have a standard deviation of about 0.010 (np.std), roughly 1 percentage point. That’s somewhat larger than the 5-fold CV standard deviation of 0.007. The 5-fold CV estimate is still the more trustworthy of the two, and not just because its std happens to be smaller. It’s built from five disjoint folds that together cover the whole dataset exactly once, systematically. The ten single-split values, by contrast, are ten separate 80/20 splits that overlap heavily. They aren’t independent measurements of the same underlying quantity, not in a structured resampling sense. A single seed’s score depends entirely on which specific examples happened to land in that one test set. CV’s reported score is already an average over five such sets. That’s exactly why it ends up both more stable and more representative than any pile of ad hoc reshuffles, however many are tried.

Exercise 13.4 [A]. A gap of zero, or effectively zero, between non-nested and nested estimates (like this chapter’s 0.9543 vs. 0.9543 result) indicates the optimism bias is negligible for this dataset/search combination. That bias comes from selecting hyperparameters on the same data used to evaluate them. Either the dataset is easy enough to separate, or the hyperparameter search space is small enough, that overfitting to the inner CV folds isn’t a serious problem here. A gap of several percentage points would indicate the opposite. The non-nested number would be meaningfully inflated by selection bias. Only the nested estimate should be reported or trusted as the honest expected performance.

Exercise 13.5 [E]. CV std tells you how much the score varies across folds of this dataset with this model. It does not tell you whether the underlying data collection process itself is representative of the deployment population. It does not tell you whether there’s target leakage baked into the features. It does not tell you whether the model will face distribution shift once deployed. A tight CV std is consistent with a reliable estimate on this data. But it says nothing about whether this data is the right data to be estimating from.

Exercise 13.6 [E]. “Regular StratifiedKFold on time-ordered data allows validation folds that sit chronologically before some of the training data within the same fold rotation. That means the model may have effectively seen future information relative to what it’s being scored on. The strong reported result is very likely optimistic and would not hold up once the model only has access to past data at deployment time. I’d ask them to rerun with TimeSeriesSplit before trusting that number.”

Exercise 13.7 [R]. Discussion only. cross_validate returns fit time and score time per fold (and can return train scores alongside validation scores, and multiple metrics simultaneously) that cross_val_score does not. This chapter’s Procedure 13.5 (comparing two models) would benefit from cross_validate’s per-fold timing when training cost, not just accuracy, is part of deciding between two otherwise similarly-accurate models.

Chapter 14 — Feature Engineering

Exercise 14.1 [W]. Tree-based models make decisions by splitting on a threshold (“is age > 35?”), and the split location adapts to whatever scale the data is already in. Multiplying a feature by 1000 just shifts where the threshold lands, without changing which points fall on which side. Linear and distance-based models combine features directly instead: weighted sums, Euclidean distances. A feature on a much larger scale mechanically dominates the combination or distance calculation, regardless of its actual predictive value, unless scaled to a comparable range first.

Exercise 14.2 [W]. (a) Ordinal encoding: small/medium/large has a genuine order. (b) Target or frequency encoding (CV-safe): 50,000 unique values feeding a linear model rules out one-hot (too many columns) and label encoding (implies a false order the linear model would misread). (c) One-hot encoding: 15 unique values, no natural order, is squarely in one-hot’s low-cardinality nominal range.

Exercise 14.3 [A]. At hour 23, hour_sin(23)=sin⁡(2π⋅23/24)≈−0.259\text{hour\_sin}(23) = \sin(2\pi \cdot 23/24) \approx -0.259, hour_cos(23)=cos⁡(2π⋅23/24)≈0.966\text{hour\_cos}(23) = \cos(2\pi \cdot 23/24) \approx 0.966. hour_sin(0)=0\text{hour\_sin}(0) = 0, hour_cos(0)=1\text{hour\_cos}(0) = 1. Euclidean distance ≈(−0.259−0)2+(0.966−1)2\approx \sqrt{(-0.259-0)^2 + (0.966-1)^2} ≈0.067+0.001≈0.26\approx \sqrt{0.067+0.001} \approx 0.26, small, correctly reflecting that hour 23 and hour 0 are only one hour apart, versus the raw integer difference of 23.

Exercise 14.4 [A]. Replacing customer_id with a column that directly (if noisily) encodes the label would very likely make the leaky-vs-safe gap grow substantially larger than 0.872 vs. 0.870. Per the Worked Example’s explanation, leakage’s damage scales with how strongly the leaking column correlates with the target once leaked. A column that’s essentially a disguised version of the label is about as strongly correlated as a feature can be. Naive (leaky) encoding would likely produce a dramatically inflated CV score that collapses on the true held-out test set, unlike the mild, honest gap measured for the weakly-predictive customer_id.

Exercise 14.5 [E]. “8,000 one-hot columns will blow up training time and memory for very little benefit. Most product IDs will appear too rarely for the model to learn anything reliable about that specific dummy column, and you’ll be feeding the model 8,000 mostly-empty columns instead of a compact signal. Frequency encoding or CV-safe target encoding captures the useful signal in one column instead.”

Exercise 14.6 [E]. A column like last_order_value_rounded (or any feature that’s derived from, or closely tracks, the order value itself after the fact) would be a strong candidate. Say such a column existed and were naively target-encoded, or even just included as a raw feature, without checking whether it’s available before the order value is known. It would very likely produce a large, disqualifying leaky-vs-safe gap, since it’s effectively leaking the label back into the features.

Exercise 14.7 [R]. Discussion only. sklearn’s built-in TargetEncoder (added in recent sklearn versions) performs its cross-fitting internally and automatically during .fit_transform(). It chooses fold splits and smoothing parameters without the user hand-writing the loop. This removes the risk of a manual implementation bug, such as an off-by-one in fold indexing, that could silently reintroduce leakage in a hand-rolled version like Procedure 14.2’s. One trap remains. On training data, call .fit_transform(), which cross-fits. Calling .fit() then .transform() on the same training data skips the cross-fitting and leaks.

Chapter 15 — Feature Selection & Dimensionality Reduction

Exercise 15.1 [W]. Feature selection removes features but keeps the survivors as-is, so “income” in the output is still literally income. Its scale, units, and meaning are unchanged, and a domain expert can interpret it directly. Feature extraction (PCA) creates new features as linear combinations of many originals. A principal component has no inherent real-world meaning of its own, only a mix of loadings across original features.

Exercise 15.2 [W]. (a) Variance threshold: filter (no model involved, judges only the feature’s own spread). (b) RFECV: wrapper (repeatedly retrains a model and cross-validates to pick a feature count). (c) Random Forest importance above the median: embedded (importance is a byproduct of fitting one model, not a separate model-agnostic step or repeated retraining).

Exercise 15.3 [A]. With k=5, SelectKBest(mutual_info_classif, k=5) keeps the 5 highest-MI-scoring features. Mutual information scores are computed once and ranked, so the top 5 are necessarily a subset of the top 10, which is itself a subset of any top-15 selection. The ranking doesn’t change with k. Only how far down that ranking you cut does. This holds only because the scores are seeded. mutual_info_classif is randomised, which is why Listing 15.3 passes random_state=42.

Exercise 15.4 [A]. Running the same cumsum(pca_full.explained_variance_ratio_) logic with the threshold raised to 0.99 gives 39 components, only 2 more than the 37 needed for 95%. That’s a smaller jump than the “climbing the last few per cent gets expensive” intuition might suggest. It’s a direct consequence of this dataset’s structure: 20 of the 50 features carry genuine signal (10 informative, 10 redundant combinations of them) and 30 are pure noise. Once PCA has captured the signal-carrying components, the remaining components are splitting up noise variance that’s spread almost evenly across the leftover dimensions, so each one adds only a sliver past the point where the real structure is already represented. A dataset with a longer tail of informative-but-individually-small directions would show a bigger 95%-to-99% jump than this one does.

Exercise 15.5 [A]. Lowering perplexity from 30 to 5 tells t-SNE to weight a much smaller local neighbourhood per point. The resulting plot typically shows tighter, more numerous, more fragmented small clusters. Local structure within digit classes becomes more visible, but the plot loses some of the smooth global separation between whole digit classes that a larger perplexity provides.

Exercise 15.6 [E]. In high dimensions, many features are noisy or redundant. A model can fit spurious patterns in that noise, which is overfitting, and irrelevant dimensions make distance-based reasoning less meaningful, the curse of dimensionality. Removing such features doesn’t just save compute. It removes the substrate the model could otherwise overfit to, which is why validation accuracy can rise, not just hold steady, after selection.

Exercise 15.7 [E]. PCA components are linear combinations of many original applicant-provided variables: income, age, credit history, and so on, all blended with different weights. A regulator asking “why was this applicant denied” cannot be given an answer in terms of “component 7,” not in any way a human applicant could act on or an auditor could verify against the original variables. Feature selection, keeping a subset of the 200 original variables via RFECV or embedded importance, preserves that direct explainability and should be recommended instead.

Exercise 15.8 [R]. Discussion only. Wattenberg et al.’s piece emphasises that t-SNE cluster sizes and inter-cluster distances in the resulting plot are not reliably meaningful. Only relative local neighbourhoods are. Applied to the digits plot: a viewer should resist reading “digit class 3’s cluster is bigger than class 7’s” or “class 1 is closer to class 4 than to class 9” as quantitative facts, even though the plot invites exactly that reading.

Chapter 16 — Model Evaluation

Exercise 16.1 [W]. A false positive is a legitimate email wrongly flagged as spam: annoying, since a real message gets buried, but rarely dangerous. A false negative is actual spam that gets through to the inbox: usually mildly annoying and cheap, though a phishing email that gets through can be costly. Between the two, false positives are generally treated as more costly for a spam filter. Losing a legitimate email, a job offer, a bill, can have real consequences, while an occasional spam message reaching the inbox is a minor inconvenience.

Exercise 16.2 [W]. (a) Recall: missing a real disease case (false negative) is the costly error in screening. (b) Precision: flagging a legitimate email as spam (false positive) is the costly error. (c) F1: with no stated cost asymmetry, a balanced single metric is the reasonable default for general comparison.

Exercise 16.3 [A]. Precision =TP/(TP+FP)=71/(71+1)=71/72≈0.9861= TP/(TP+FP) = 71/(71+1) = 71/72 \approx 0.9861. Recall =TP/(TP+FN)=71/(71+1)=71/72≈0.9861= TP/(TP+FN) = 71/(71+1) = 71/72 \approx 0.9861. Both match the reported figures. In this case, precision and recall come out numerically equal because FP and FN happen to both equal 1 on this particular test set.

Exercise 16.4 [A]. A model predicting “fraud” for every transaction in the 0.1%-fraud scenario would be correct only on the 0.1% that are fraud, giving accuracy ≈0.001\approx 0.001 (0.1%), catastrophically low. Precision would be TP/(TP+FP)=0.001TP/(TP+FP) = 0.001, since only 0.1% of its “fraud” calls are actually fraud. Recall would be 1.01.0: it catches every real fraud case, trivially, by flagging everything. This is the mirror-image failure mode to the “always predict not-fraud” model: perfect recall bought at the cost of near-total precision collapse.

Exercise 16.5 [E]. “99.9% accuracy on a fraud dataset where fraud is rare is close to what you’d get by predicting ‘not fraud’ every single time. Accuracy alone can’t distinguish a good model from a completely useless one under this kind of imbalance. Before agreeing it’s ready, I’d want to see recall and precision (or PR-AUC) specifically on the fraud class, since those are the numbers that actually say whether it catches real fraud.”

Exercise 16.6 [E]. Random Forests are a common source of S-shaped miscalibration, under-confident at the extremes. A predict_proba of 0.75 might correspond to an observed positive rate closer to 0.9. Using such probabilities directly to set premiums risks systematically mispricing risk at the extremes: charging too little for the genuinely high-risk cases, and too much for the lowest-risk ones. Before trusting the numbers, plot a reliability diagram (Procedure 16.4). If it deviates from the diagonal, apply CalibratedClassifierCV with isotonic regression before using the probabilities for pricing.

Exercise 16.7 [R]. Discussion only. Davis and Goadrich’s core claim is that ROC curves can present an overly optimistic picture of classifier performance specifically under class imbalance. The false positive rate, the ROC curve’s x-axis, is normalised by the large number of true negatives, making even a fairly high absolute count of false positives look small. Precision, the PR curve’s y-axis, is by contrast directly sensitive to that same false positive count relative to total positive predictions. This is exactly why this chapter recommends PR-AUC over ROC-AUC for rare-event detection: PR-AUC doesn’t get the same optimistic distortion from an abundance of true negatives.

Chapter 17 — Hyperparameter Optimisation

Exercise 17.1 [W]. 55=3,1255^5 = 3{,}125 because grid search enumerates every combination across all 5 hyperparameters’ 5 values each, with no mechanism to notice or exploit the fact that only some of them matter. It treats every dimension as equally worth exhaustive coverage. Even if only 2 of the 5 affect performance, grid search still multiplies out all 5 dimensions blindly. It has no way to detect in advance which ones are the unimportant ones.

Exercise 17.2 [W]. (a) Grid search is reasonable: 2 hyperparameters and sub-second training keeps even exhaustive coverage cheap, and grid search’s full, reproducible coverage has no real downside at this scale. (b) Bayesian optimisation (Optuna) is the better default: 8 hyperparameters and 20-minute training runs make each evaluation expensive, exactly the regime where sample efficiency (fewer wasted trials) matters most.

Exercise 17.3 [A]. The gap is 0.8942−0.8933=0.00090.8942 - 0.8933 = 0.0009 (0.09 percentage points) in random search’s favour. It’s small, but it isn’t noise. It traces directly back to the grid’s construction. With only two values per dimension across six hyperparameters, every one of grid search’s 64 combinations sits at a corner of the search space: always the minimum or maximum subsample, max_depth, and so on. It can never place a trial in the interior of any dimension. Random search samples continuously across the same six ranges, so it can land on interior values the grid structurally excludes. It does: its winning configuration used max_depth=6 and subsample~0.88, values no combination in the 64-corner grid could ever produce.

Exercise 17.4 [A]. Sketch:

def objective(trial):
    params = {...}
    model = xgb.XGBClassifier(
        **params, random_state=42, verbosity=0
    )
    for round_num in range(0, params['n_estimators'], 10):
        # train incrementally / use eval_set with early rounds
        # validation AUC after this many rounds
        intermediate_auc = ...
        trial.report(intermediate_auc, step=round_num)
        if trial.should_prune():
            raise optuna.exceptions.TrialPruned()
    return final_auc

study = optuna.create_study(
    direction='maximize',
    pruner=optuna.pruners.MedianPruner(
        n_startup_trials=5, n_warmup_steps=10
    ),
)

Trials tracking below the median of previous trials at the same step get abandoned early, saving the remaining boosting rounds’ compute for more promising configurations.

Exercise 17.5 [E]. “Optuna’s sample efficiency pays off when you use it as a sample-efficient method: stop once it finds a good-enough configuration, rather than forcing it through the same trial budget as a cruder strategy. In this chapter’s matched comparison, Optuna needed only 5 trials (5.7 seconds) to reach the CV AUC that random search needed its full 64 trials (18.5 seconds) to find. It went on to find the best score of the three strategies by the time its own 64 trials were up. But that full run took 171.5 seconds, more than the other two combined, because nobody told it to stop once it had already won. ‘Always use Optuna’ is right about its accuracy and its trial efficiency. It’s wrong if the trial budget gets copied from a strategy that doesn’t need to be stopped early.”

Exercise 17.6 [E]. “Was that 0.97 validation AUC measured on the same folds used during the hyperparameter search, or on a held-out test set that never participated in tuning? If it’s the former, the number is optimistic by construction: the search process selected for whichever configuration happened to score best on those specific folds, and the honest number is whatever a truly untouched test set reports.”

Exercise 17.7 [R]. Discussion only. Bergstra and Bengio’s argument is strongest in higher-dimensional search spaces, where only a small effective subset of hyperparameters meaningfully affects the objective, and grid search is additionally restricted to just a couple of values per dimension. The more total dimensions, and the coarser grid search’s coverage of each one, the more its exhaustive-but-uniform corner sampling is wasted relative to a method that can reach the interior. This chapter’s own comparison used six hyperparameters with a grid restricted to two boundary values each, exactly that regime. It showed the predicted gap. Random search beat grid on both CV and test AUC, and Optuna beat both by using its trial history to guide where it sampled next. It also showed a side of the argument Bergstra and Bengio don’t dwell on: Bayesian optimisation’s real advantage over random search shows up in trials needed, not in a matched-budget wall clock.

Chapter 18 — Regularisation

Exercise 18.1 [W]. Ridge’s penalty (∑wi2\sum w_i^2) has a smooth, round constraint boundary with no corners. The optimal solution under gradient descent settles at whatever point balances fit against penalty smoothly: weights shrink continuously towards zero but rarely land exactly on it. Lasso’s penalty (∑∣wi∣\sum |w_i|) has a diamond-shaped constraint region with corners sitting exactly on the coordinate axes. The loss-minimising solution frequently lands precisely on one of those corners, which corresponds to a coefficient of exactly zero.

Exercise 18.2 [W]. This is a variance problem: training error is low (1.2) but test error is much higher (3.1), the classic large train/test gap signature of overfitting (high variance), not underfitting (which would show high error on both). Any of Ridge, Lasso, Elastic Net, dropout (if a neural net), or early stopping would help.

Exercise 18.3 [A]. Unregularised ratio: 1.912/1.412≈1.351.912/1.412 \approx 1.35. Test error is about 35% higher than training error, a real but modest overfitting gap given how small this dataset is. Ridge ratio: 1.461/1.819≈0.801.461/1.819 \approx 0.80. Test error is below training error. A ratio below 1.0 shows the model isn’t just avoiding overfitting. It’s generalising better than its own training performance would suggest. That can happen when the test split happens to be marginally “easier,” or when the regularised model is closer to the true underlying function than to the noisy training sample.

Exercise 18.4 [A]. The true data-generating process is y=0.5x2−x+2y = 0.5x^2 - x + 2 plus noise: a quadratic, needing only an x2x^2 term and an x1x^1 term (plus the intercept, handled separately) to represent exactly. A degree-15 polynomial expansion has 15 non-intercept terms. Lasso zeroing 11 of them and keeping 4 is broadly consistent with the model identifying that only a couple of low-degree terms carry the real signal. The eliminated terms exist only to fit noise, which Lasso’s sparsity mechanism is specifically built to remove. The 2 extra nonzero terms beyond the ideal count aren’t a contradiction. With only 42 training points, cross-validation picked whichever alpha minimised validation error, not whichever alpha recovered the exact ground-truth term count. A small amount of residual noise-fitting surviving alongside the real quadratic signal is expected.

Exercise 18.5 [E]. Setting alpha=100 without cross-validating risks landing far past the sweet spot into the underfitting regime. An arbitrarily large, unvalidated α\alpha can suppress real signal along with noise. The result is a model that’s “safe” from overfitting only because it’s no longer capturing the pattern at all, with both training and test performance suffering together. The Trade-offs section’s guidance is explicit: find the sweet spot via cross-validated search, not by picking a large number and hoping.

Exercise 18.6 [E]. The feature in the thousands will have its weight naturally driven small by ordinary least-squares fitting, to keep its large-magnitude contribution to predictions reasonable. The L2/L1 penalty, which acts on raw weight magnitude, barely touches it as a result. The 0-to-1-scaled feature’s weight, meanwhile, will naturally be larger to have any effect at all, so it absorbs a disproportionate share of the regularisation penalty. The practical effect: the large-scale feature is under-regularised, and the small-scale feature is over-regularised, purely as an artefact of scale rather than actual relevance. That’s exactly why scaling must always precede regularisation.

Exercise 18.7 [R]. Discussion only. In the Gaussian-prior interpretation, the prior’s variance σ2\sigma^2 and Ridge’s α\alpha are inversely related. A smaller prior variance expresses stronger belief that weights should be close to zero, which mathematically corresponds to a larger penalty (larger α\alpha) pulling the MAP (maximum a posteriori) estimate towards zero more forcefully. A very small prior variance is a strong prior belief in small weights, exactly mirroring a large α\alpha’s strong regularisation pressure.

Chapter 19 — Unsupervised Learning

Exercise 19.1 [W]. K-Means’s algorithm requires KK centroids to initialise and assign points to before it can even begin iterating. The number of centroids is a structural input to the procedure, not something it discovers. DBSCAN instead defines clusters implicitly, as whatever dense regions emerge from the eps/min_samples density criteria applied to the actual data. However many dense regions exist is however many clusters result, with no upfront count required.

Exercise 19.2 [W]. (a) K-Means: large, roughly spherical, well-separated segments are exactly K-Means’s best-case assumption, and it scales better than DBSCAN on large data. (b) DBSCAN: an unknown cluster count with irregular shapes and genuine noise is exactly the case K-Means structurally cannot handle (fixed K, spherical assumption, no built-in outlier concept) and DBSCAN is built for.

Exercise 19.3 [A]. Silhouette score measures how well each point fits its assigned cluster relative to the next-nearest cluster, using whatever clustering was produced. It has no access to the true crescent-shaped structure, and cannot detect that K-Means drew the wrong kind of boundary. A K-Means clustering on the moons data could report a moderate-to-decent silhouette score for the (wrong) roughly-circular split it found. That’s precisely because silhouette only asks “is this point closer, on average, to the points in its own cluster than to the points in the nearest other one,” which K-Means’s convex clusters can satisfy reasonably well even while completely missing the true crescent shapes. Silhouette validates internal consistency with the algorithm’s own assumptions, not correctness against ground truth.

Exercise 19.4 [A]. With 27 flagged, precision 0.852 means 27×0.852≈2327 \times 0.852 \approx 23 true positives, so false positives =27−23=4= 27 - 23 = 4. Recall 0.920 on 25 true anomalies means 25×0.920=2325 \times 0.920 = 23 true positives (consistent), so false negatives =25−23=2= 25 - 23 = 2. This run made more false positives (4) than false negatives (2): it over-flagged slightly more than it missed, though both error types are present.

Exercise 19.5 [E]. “How well do those 6 clusters hold up when you show them to someone who knows these customers? Do the groups correspond to real, distinct behaviours, or just to a geometrically tidy split of the feature space? A good silhouette score tells you the clusters are internally compact and well-separated; it says nothing about whether they’re the useful 6 groups for your business decisions.”

Exercise 19.6 [E]. Setting contamination to whatever value produces the “cleanest-looking” separation is circular, because it’s using the model’s own output to justify the input that produced that output. Instead of contamination reflecting an actual prior belief about how much of the data is truly anomalous, it’s being reverse-engineered from a desired visual result. This risks hiding real anomalies (if the “clean” contamination is set too low), or manufacturing fake ones (if set too high), rather than reflecting genuine fraud prevalence.

Exercise 19.7 [R]. Discussion only. The Isolation Forest paper’s key complexity claim is that it achieves roughly linear time complexity and constant memory with respect to dataset size. That’s because it doesn’t need to compute pairwise distances or density estimates the way distance-based methods (like LOF or One-Class SVM) do: isolation only requires random splits, not distance comparisons between all pairs of points. This directly explains this chapter’s “scales well, works in high dimensions” summary. Avoiding pairwise distance computation is exactly what lets it scale where density- or distance-based alternatives struggle.

Chapter 20 — Time Series Analysis

Exercise 20.1 [W]. Earlier chapters’ datasets were assumed independent and identically distributed. Each row’s order carries no information, so shuffling before splitting doesn’t change what the model can learn. Time series data has the opposite property by definition: today’s value depends on yesterday’s. A random split lets some training rows sit chronologically after some test rows, letting the model implicitly “see” future information relative to what it’s being scored on.

Exercise 20.2 [W]. A slowly-decaying ACF (no sharp cutoff) is the signature of non-stationarity: the series’ statistical properties are drifting over time rather than settling into a stable pattern. The action it suggests is differencing the series (and re-running the ADF test) before attempting to fit ACF/PACF-based model orders or an ARIMA model at all.

Exercise 20.3 [A]. Rejecting the null hypothesis at p≈3×10−27<0.05p \approx 3 \times 10^{-27} < 0.05 means there’s overwhelming statistical evidence against “this series has a unit root,” i.e. is non-stationary. The differenced series can be treated as stationary, satisfying the assumption ARIMA-family models need. The raw series’ p=0.682p=0.682, far above 0.05, means the null hypothesis of non-stationarity cannot be rejected. That’s consistent with the visible upward trend that was deliberately built into the synthetic data.

Exercise 20.4 [A]. Naive MAE (4.602) is lower than ARIMA MAE (7.228) by (7.228−4.602)/7.228≈36.3%(7.228-4.602)/7.228 \approx 36.3\%: naive outperforms ARIMA by roughly 36% on this metric. A longer horizon (say, 180 or 365 days) would more plausibly reverse this result. “Repeat the last value” becomes a progressively worse assumption the further out the trend and seasonal cycle are expected to move. ARIMA’s explicit trend/autoregressive structure has more room to demonstrate a genuine advantage once the naive baseline’s flat-line assumption clearly breaks down.

Exercise 20.5 [E]. “Did you compare that 7.2 MAE against a naive baseline, just repeating the last observed value for all 28 days? This chapter’s own measurement found exactly that comparison flipping the expected result: a naive baseline beat a reasonably-specified ARIMA model at this horizon. An MAE number in isolation, without the baseline comparison, doesn’t tell you whether the model is adding value.”

Exercise 20.6 [E]. Computing .rolling(7).mean() without .shift(1) first means the rolling window for row tt includes row tt’s own value. The feature meant to summarise “the past 7 days” includes today’s target value baked into its own input. That’s a direct form of leakage: the model gets to see (an average containing) the answer as part of its own input feature.

Exercise 20.7 [R]. Discussion only. Hyndman and Athanasopoulos discuss measures like MASE (Mean Absolute Scaled Error), which scales errors relative to a naive/seasonal-naive baseline’s own error rather than relative to the raw target values. It’s designed specifically to allow comparison across series of very different scales, and to remain well-behaved when the series can be at or near zero, exactly the situation this chapter flags as breaking MAPE.

online /books/classical-data-science/appendix-b-answers Newcastle, UK v0.1.0 // rss