◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Chapter 18 quiz · 23 questions

Regularisation

Tests how to read learning curves, compute and compare L1, L2 and Elastic Net penalties, set alpha and C in the right direction, and use dropout and early stopping correctly.

Based on chapter 18. Every answer comes with an explanation and a link back to the section.

/Q18

Quiz

Question 1 of 23

Bias-variance and learning-curve diagnosis Level 1
Progress

On a learning curve, training and validation error both converge to a similar, high error as the training set grows. What does this show, and what is the fix?

Choose one
/02

练习From the book

7 exercises

The chapter's own exercises are open questions. Try one, then compare with the model answer.

Exercise 18.1 [W]

Explain in your own words why Ridge shrinks weights towards zero without ever setting them exactly to zero, while Lasso can.

Show the model answer

Ridge's penalty (∑wi2\sum w_i^2) has a smooth, round constraint boundary with no corners. The optimal solution under gradient descent settles at whatever point balances fit against penalty smoothly: weights shrink continuously towards zero but rarely land exactly on it. Lasso's penalty (∑∣wi∣\sum |w_i|) has a diamond-shaped constraint region with corners sitting exactly on the coordinate axes. The loss-minimising solution frequently lands precisely on one of those corners, which corresponds to a coefficient of exactly zero.

Exercise 18.2 [W]

A model shows training RMSE 1.2 and test RMSE 3.1. State whether this is a bias problem or a variance problem, and name one regularisation technique that would help.

Show the model answer

This is a variance problem: training error is low (1.2) but test error is much higher (3.1), the classic large train/test gap signature of overfitting (high variance), not underfitting (which would show high error on both). Any of Ridge, Lasso, Elastic Net, dropout (if a neural net), or early stopping would help.

Exercise 18.3 [A]

Using Listing 18.7's measured numbers, compute the ratio of test RMSE to train RMSE for the unregularised model (1.912/1.412) and for the Ridge model (1.461/1.819), and explain what a ratio below 1.0 shows about the Ridge model's fit.

Show the model answer

Unregularised ratio: 1.912/1.412≈1.351.912/1.412 \approx 1.35. Test error is about 35% higher than training error, a real but modest overfitting gap given how small this dataset is. Ridge ratio: 1.461/1.819≈0.801.461/1.819 \approx 0.80. Test error is below training error. A ratio below 1.0 shows the model isn't just avoiding overfitting. It's generalising better than its own training performance would suggest. That can happen when the test split happens to be marginally "easier," or when the regularised model is closer to the true underlying function than to the noisy training sample.

Exercise 18.4 [A]

Using Listing 18.4's pattern, explain why Lasso zeroing 11 of 15 polynomial coefficients is broadly consistent with the true underlying relationship being quadratic (degree 2, i.e. needing only 2 non-trivial polynomial terms beyond the intercept), even though a few extra nonzero terms remain, likely absorbing noise from a training set of only 42 points.

Show the model answer

The true data-generating process is y=0.5x2−x+2y = 0.5x^2 - x + 2 plus noise: a quadratic, needing only an x2x^2 term and an x1x^1 term (plus the intercept, handled separately) to represent exactly. A degree-15 polynomial expansion has 15 non-intercept terms. Lasso zeroing 11 of them and keeping 4 is broadly consistent with the model identifying that only a couple of low-degree terms carry the real signal. The eliminated terms exist only to fit noise, which Lasso's sparsity mechanism is specifically built to remove. The 2 extra nonzero terms beyond the ideal count aren't a contradiction. With only 42 training points, cross-validation picked whichever alpha minimised validation error, not whichever alpha recovered the exact ground-truth term count. A small amount of residual noise-fitting surviving alongside the real quadratic signal is expected.

Exercise 18.5 [E]

A colleague sets alpha=100 for a Ridge model "to be safe against overfitting" without cross-validating it. Using this chapter's Trade-offs section, explain what's likely to go wrong.

Show the model answer

Setting alpha=100 without cross-validating risks landing far past the sweet spot into the underfitting regime. An arbitrarily large, unvalidated α\alpha can suppress real signal along with noise. The result is a model that's "safe" from overfitting only because it's no longer capturing the pattern at all, with both training and test performance suffering together. The Trade-offs section's guidance is explicit: find the sweet spot via cross-validated search, not by picking a large number and hoping.

Exercise 18.6 [E]

A teammate regularises a model with features on wildly different scales (one column in the thousands, another between 0 and 1) without scaling first. Using Procedure 18.3's common mistake, predict what will happen to the two features' effective regularisation strength.

Show the model answer

The feature in the thousands will have its weight naturally driven small by ordinary least-squares fitting, to keep its large-magnitude contribution to predictions reasonable. The L2/L1 penalty, which acts on raw weight magnitude, barely touches it as a result. The 0-to-1-scaled feature's weight, meanwhile, will naturally be larger to have any effect at all, so it absorbs a disproportionate share of the regularisation penalty. The practical effect: the large-scale feature is under-regularised, and the small-scale feature is over-regularised, purely as an artefact of scale rather than actual relevance. That's exactly why scaling must always precede regularisation.

Exercise 18.7 [R]

Read the Bishop (Pattern Recognition and Machine Learning) Ch. 3 treatment of L2 as a Gaussian prior. Explain, in your own words, how choosing a smaller prior variance corresponds to a larger α\alpha in the Ridge loss function.

Show the model answer

Discussion only. In the Gaussian-prior interpretation, the prior's variance σ2\sigma^2 and Ridge's α\alpha are inversely related. A smaller prior variance expresses stronger belief that weights should be close to zero, which mathematically corresponds to a larger penalty (larger α\alpha) pulling the MAP (maximum a posteriori) estimate towards zero more forcefully. A very small prior variance is a strong prior belief in small weights, exactly mirroring a large α\alpha's strong regularisation pressure.

online /books/classical-data-science/quiz/18 Newcastle, UK v0.1.0 // rss