Tests computing and choosing metrics from confusion matrices, thresholds, ranked scores, probabilities and regression errors, and spotting when a good-looking number misleads.
Based on chapter 16. Every answer comes with an explanation and a link back to the section.
/Q16
Quiz
Question 1 of 25
The confusion matrix, accuracy and the accuracy paradox Level 1
Progress
A fraud model flags a legitimate transaction as fraud (fraud is the positive class). Which cell of the confusion matrix does this land in?
/02
练习From the book
7 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 16.1 [W]
Using the confusion matrix definitions, explain in one sentence each what a false positive and a false negative mean for a spam filter, and state which one is more costly there.
Show the model answer
A false positive is a legitimate email wrongly flagged as spam: annoying, since a real message gets buried, but rarely dangerous. A false negative is actual spam that gets through to the inbox: usually mildly annoying and cheap, though a phishing email that gets through can be costly. Between the two, false positives are generally treated as more costly for a spam filter. Losing a legitimate email, a job offer, a bill, can have real consequences, while an occasional spam message reaching the inbox is a minor inconvenience.
Exercise 16.2 [W]
State which metric (precision, recall, or F1) is most appropriate for: (a) a disease-screening test; (b) a spam filter; (c) a general-purpose classifier comparison with no stated cost asymmetry.
Show the model answer
(a) Recall: missing a real disease case (false negative) is the costly error in screening. (b) Precision: flagging a legitimate email as spam (false positive) is the costly error. (c) F1: with no stated cost asymmetry, a balanced single metric is the reasonable default for general comparison.
Exercise 16.3 [A]
Using Listing 16.5, compute precision and recall by hand from TN=41, FP=1, FN=1, TP=71, and confirm they match the reported 0.9861 figures.
Show the model answer
Precision =TP/(TP+FP)=71/(71+1)=71/72≈0.9861. Recall =TP/(TP+FN)=71/(71+1)=71/72≈0.9861. Both match the reported figures. In this case, precision and recall come out numerically equal because FP and FN happen to both equal 1 on this particular test set.
Exercise 16.4 [A]
Using this chapter's hypothetical 0.1%-fraud scenario, compute the accuracy of a model that predicts "fraud" for every transaction instead (the opposite extreme), and state its precision and recall.
Show the model answer
A model predicting "fraud" for every transaction in the 0.1%-fraud scenario would be correct only on the 0.1% that are fraud, giving accuracy ≈0.001 (0.1%), catastrophically low. Precision would be TP/(TP+FP)=0.001, since only 0.1% of its "fraud" calls are actually fraud. Recall would be 1.0: it catches every real fraud case, trivially, by flagging everything. This is the mirror-image failure mode to the "always predict not-fraud" model: perfect recall bought at the cost of near-total precision collapse.
Exercise 16.5 [E]
A customer reports "our fraud model has 99.9% accuracy" as evidence it's ready to ship. Using this chapter's Worked Example, write the two or three sentences you'd say before agreeing.
Show the model answer
"99.9% accuracy on a fraud dataset where fraud is rare is close to what you'd get by predicting 'not fraud' every single time. Accuracy alone can't distinguish a good model from a completely useless one under this kind of imbalance. Before agreeing it's ready, I'd want to see recall and precision (or PR-AUC) specifically on the fraud class, since those are the numbers that actually say whether it catches real fraud."
Exercise 16.6 [E]
A Random Forest's predict_proba output is being used directly to set insurance premiums. Using this chapter's Calibration section, explain what could go wrong and what you'd check before trusting those numbers.
Show the model answer
Random Forests are a common source of S-shaped miscalibration, under-confident at the extremes. A predict_proba of 0.75 might correspond to an observed positive rate closer to 0.9. Using such probabilities directly to set premiums risks systematically mispricing risk at the extremes: charging too little for the genuinely high-risk cases, and too much for the lowest-risk ones. Before trusting the numbers, plot a reliability diagram (Procedure 16.4). If it deviates from the diagonal, apply CalibratedClassifierCV with isotonic regression before using the probabilities for pricing.
Exercise 16.7 [R]
Read the Davis and Goadrich (2006) paper's abstract. Identify the specific claim it makes about ROC curves under class imbalance, and explain how it relates to this chapter's PR-AUC recommendation for rare-event detection.
Show the model answer
Discussion only. Davis and Goadrich's core claim is that ROC curves can present an overly optimistic picture of classifier performance specifically under class imbalance. The false positive rate, the ROC curve's x-axis, is normalised by the large number of true negatives, making even a fairly high absolute count of false positives look small. Precision, the PR curve's y-axis, is by contrast directly sensitive to that same false positive count relative to total positive predictions. This is exactly why this chapter recommends PR-AUC over ROC-AUC for rare-event detection: PR-AUC doesn't get the same optimistic distortion from an abundance of true negatives.