Tests whether you can do the maths of training by hand. Derivatives and gradients, gradient descent steps, learning-rate limits, the chain rule through a tiny network, loss values, mini-batch counts, and momentum and Adam updates.
Based on chapter 4. Every answer comes with an explanation and a link back to the section.
/Q4
Quiz
Question 1 of 26
Derivatives, partial derivatives and gradients Level 1
Progress
You are training a model with one weight w. At the current value, the derivative of the loss is dwdL=−3. For a small step, which change to w lowers the loss?
/02
练习From the book
8 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 4.1 [W]
Explain in your own words why ∇f(w) points in the direction of steepest ascent, and why gradient descent then steps in the negative gradient direction.
Show the model answer
The gradient's each component ∂f/∂wi tells you how f changes as wi increases. By construction, moving in the direction of that vector increases f fastest. That's what "steepest ascent" means. Since the goal in training is to decrease the loss, the correct move is the exact opposite direction, −∇f(w). That's what the gradient descent update rule (w←w−α∇f(w)) implements.
Exercise 4.2 [W]
A training run's loss oscillates wildly between epochs 5 and 40, then goes to NaN at epoch 41. Using Listing 4.2's diagnostic table, name the most likely cause and the first thing to try.
Show the model answer
Oscillation followed by a NaN is the "learning rate too high" pattern carried to its extreme. The steps have been overshooting the minimum by a growing margin each time: each overshoot lands further up the opposite wall of the loss surface than the last. Eventually a step produces a value large enough to overflow floating point. The first thing to try is reducing the learning rate by roughly 10x, since Listing 4.2's table specifically maps both "oscillating" and "NaN or infinity" back to the same root cause.
Exercise 4.3 [A]
Using Listing 4.6, rerun gradient descent on f(x)=x2 starting at x=3.0 with learning rate α=1.1 instead of 0.2. Report what happens to x over the first 5 steps. Explain it in terms of the (1−2α) contraction factor mentioned in the Worked Example.
Show the model answer
At α=1.1, the per-step contraction factor is (1−2α)=1−2.2=−1.2, whose magnitude exceeds 1. So instead of shrinking towards zero, x grows in magnitude every step and flips sign each time. Starting at x=3.0, the sequence runs approximately 3.0→−3.6→4.32→−5.184→6.221→−7.465 over the first 5 steps, diverging geometrically. This is the 1D version of the same "too large" divergence shown for linear regression in the Worked Example's Stage 3. The general rule for f(x)=x2 is that gradient descent converges only for 0<α<1 (i.e. ∣1−2α∣<1).
Exercise 4.4 [A]
Using Listing 4.7, rerun training with learning_rate=0.5 instead of 0.1 for 200 epochs, and report the epoch at which the loss first comes within 1% of the optimal loss (406.52), compared to the ~17 epochs it took at α=0.1.
Show the model answer
At α=0.5, five times the original 0.1, the loss reaches the optimum after just 1 epoch: w = [-0.689864, 80.790445], loss 406.5247, an exact match to the closed-form answer, and it stays there for every epoch after. That's not an approximation rounding down to "epoch 1"; it's exact. This particular dataset's Hessian, N2X⊤X, works out to exactly 2I: the standardised feature column is, by construction, uncorrelated with the constant bias column, so the two parameters have identical curvature and no cross-term. A learning rate of α=0.5 satisfies α⋅2=1 exactly, which zeroes out (I−αH) and collapses gradient descent to the closed-form solution in a single step, regardless of the starting point. This is a property of this specific, well-conditioned, standardised, two-parameter problem, not a general rule for α=0.5. It's a sharper version of the chapter's "too small wastes epochs" point. On a well-conditioned problem, a larger, still-stable learning rate does not just converge faster. It can hit the exact optimum directly. 0.5 remains well inside the stable range for this loss surface, short of the divergence seen at 1.5.
Exercise 4.5 [A]
Using Listing 4.8 as a starting point, add a fourth learning rate of your choosing between 0.1 and 1.5, run it for 100 epochs. Classify it as behaving more like "just right" or "too large," based on where the loss lands by epoch 30.
Show the model answer
Answers vary by the exact value chosen, but on this specific, standardised, two-parameter problem the dynamics are governed exactly by the contraction factor ∣1−2α∣ (per Exercise 4.4's finding that this problem's Hessian is exactly 2I). Any α between 0 and 1 converges, at different speeds. α=0.3 or α=0.7 (contraction 0.4) reach within 1% of optimal by epoch 5; α=0.2 or α=0.8 (contraction 0.6) by epoch 8; α=0.05 (contraction 0.9, close to the chapter's own α=0.1) takes about 36 epochs. At exactly α=1.0, the contraction factor is ∣1−2∣=1: the loss neither grows nor shrinks, oscillating at a fixed distance from the optimum forever rather than converging. Only once α exceeds 1.0 does the contraction factor exceed 1 and the loss diverge, exactly as α=1.1 does in Exercise 4.3. So the real dividing line between "just right" and "too large" on this problem sits at α=1.0, not a fuzzy "1.0-1.2" range. The qualitative marker of "too large" is loss increasing rather than decreasing over the run, not merely converging slowly.
Exercise 4.6 [E]
The chapter states that Adam's adaptive per-parameter scaling makes it "less sensitive to the initial choice of learning rate than SGD." Propose an experiment (you don't need to run it) that would test this claim directly against plain SGD. State what result would falsify it.
Show the model answer
A reasonable experiment: train the same model twice, once with plain SGD and once with Adam. Sweep the initial learning rate over several orders of magnitude (e.g. 10−5 to 100) for each. Plot final loss (or validation accuracy) against the chosen learning rate for both optimisers. The claim predicts Adam's curve should stay flat and near-optimal across a wide band of learning rates, while SGD's curve should be sharply peaked: good only in a narrow band, diverging above it and painfully slow below it. The claim would be falsified if Adam's performance instead degraded about as sharply as SGD's outside some narrow band, i.e. if the two curves had comparably narrow "good" regions.
Exercise 4.7 [E]
A colleague proposes using scipy.optimize.minimize with method='L-BFGS-B' to fine-tune a 50-million-parameter neural network because "it converged in 2 iterations on our small test problem." Using this chapter's argument on why scipy's solvers don't suit neural network training, write the two or three sentences you'd say to redirect them.
Show the model answer
A reasonable response: memory isn't the problem. L-BFGS-B keeps only the last few gradient differences, about 10 pairs, so at 50 million parameters it needs roughly 20 vectors × 50M × 8 bytes, about 8 GB. That fits on one GPU. The problem is the gradients. Its 2-iteration speed came from exact, full-batch gradients on a smooth convex problem. A neural network is trained on mini-batches, because the training data rarely fits in memory for a full pass, and mini-batch gradients are noisy. That noise breaks L-BFGS-B's line search and its curvature estimates. For SGD and Adam the same noise helps, because it pushes the optimiser past saddle points on a non-convex loss surface. So the fast convergence you saw doesn't transfer, and mini-batch SGD or Adam is the right tool here. The square-size Hessian argument applies to exact Newton's method, which would need about 4,000 GB (3,725 GiB) at only 1 million parameters. It doesn't apply to L-BFGS.
Exercise 4.8 [R]
Read Goh's "Why Momentum Really Works" (cited above). Explain, in terms distinct from this chapter's "rolling ball" intuition, why momentum specifically helps in loss surfaces shaped like a narrow curved valley (an ill-conditioned quadratic). Explain how that differs from the plain "escapes local minima via noise" story told about SGD.
Show the model answer
Discussion only. No single answer. Goh's piece frames momentum's benefit less as "escaping local minima via averaged-out noise" (the SGD story), and more as correcting for ill-conditioning. When a loss surface curves much more steeply in one direction than another (a narrow, elongated valley), plain gradient descent zig-zags across the narrow axis, while making frustratingly slow progress along the long axis. A single learning rate has to be small enough not to diverge on the steep direction, which caps progress on the shallow one. Momentum's accumulated velocity partially cancels the oscillation across the narrow axis, since opposing updates average out. It reinforces the consistent direction along the long axis, since aligned updates add up. That's a curvature-shape argument, distinct from and complementary to the "noise helps escape bad minima" story usually told about SGD. Worth noting the two mechanisms address different pathologies (conditioning vs. non-convexity).