Tests turning a vague business ask into a precise ML problem statement: stakeholders and goal statements, labels, units, horizons and leakage, business success metrics, cost-based thresholds, and choosing the problem type.
Based on chapter 2. Every answer comes with an explanation and a link back to the section.
/Q2
Quiz
Question 1 of 24
Stakeholders and the three-prong goal statement Level 1
Progress
On a churn project, the Director of Retention must sign off before the model goes live and judges whether it solved the problem. Which of the three stakeholder groups is this?
/02
练习From the book
7 exercises
The chapter's own exercises are open questions. Try one, then compare with the model answer.
Exercise 2.1 [W]
A customer says "we want to predict customer satisfaction." Explain in one sentence why this is not yet a usable label, and propose a candidate label that would be.
Show the model answer
"Customer satisfaction" is not measurable or tied to a specific event. It's a survey construct, not something observable in transactional data at a given moment. A usable label ties it to something concrete and time-bound, e.g. submitted_cancellation_request_within_30d or nps_score_below_6_next_survey.
Exercise 2.2 [W]
For each of the following, name the correct entry from the Section 2.2.5 taxonomy: (a) grouping customers into segments with no target variable; (b) predicting the dollar value of a customer's next order; (c) predicting whether a machine will fail in the next 48 hours.
Show the model answer
(a) Clustering: no target variable, discovering natural groups. (b) Regression: predicting a continuous dollar quantity. (c) Binary classification: fail or not fail within a fixed 48-hour horizon. Predicting which failure mode is a multi-class extension.
Exercise 2.3 [A]
Using the churn dataset schema from Listing 2.4, propose a new candidate feature days_since_last_price_increase. Using Procedure 2.5, determine whether it is safe to use. Justify your answer.
Show the model answer
days_since_last_price_increase is safe only if the price increase is known and dated before the prediction point for every row. It can't be retroactively determined by something that happens after prediction time, such as a price change made in response to early churn signals. If price changes are scheduled independently of individual customer behaviour, the feature is safe. If the business sometimes offers reactive discounts to at-risk customers (which would correlate with the label), it is a leakage risk. It should be excluded, or replaced with a feature computed strictly from data available before the churn-risk window.
Exercise 2.4 [A]
Using Listing 2.5's probabilities and Listing 2.3's sweep, verify the total business cost at thresholds 0.3, 0.5, and 0.7 with fp_cost=5, fn_cost=200. State which threshold minimises cost, and why.
Show the model answer
On the class-weighted logistic regression from Listing 2.5, the three thresholds cost $9,345 (0.3: 1,869 false positives, 0 false negatives), $18,255 (0.5: 651 false positives, 75 false negatives), and $24,785 (0.7: 37 false positives, 123 false negatives). 0.3 wins, and by a wide margin. With fp_cost=5 and fn_cost=200, a single missed churner costs as much as 40 false alarms, so the cost-minimising threshold is pulled towards the low end of the swept range even though it means flagging almost every customer in the test set (this model's probabilities never drop below about 0.32, so 0.3 is close to "flag everyone"). The full 50-point sweep confirms it: the true minimum sits at a threshold of 0.345 ($9,185), just above 0.3, not at some higher, more conservative-looking value.
Exercise 2.5 [E]
The chapter argues accuracy is rarely the right metric. Construct a scenario, distinct from fraud or churn, where accuracy is an acceptable metric. Explain why the class-imbalance argument doesn't apply there.
Show the model answer
A model routing internal support tickets to one of a few roughly equal-sized teams is a reasonable case for accuracy. Misrouting costs a short reassignment step, not a missed high-stakes event. Classes are close to balanced, and the cost of a false positive/negative is symmetric and low. The imbalance argument specifically targets skewed class distributions with asymmetric costs (fraud, churn, disease), where accuracy can be trivially gamed by always predicting the majority class. Symmetric-cost, balanced-class problems don't have that failure mode.
Exercise 2.6 [E]
A customer asks for "AI to detect anomalies in manufacturing sensor data" but has zero historical examples of past failures. Using the ML Suitability Checklist in Appendix A, decide whether ML is appropriate here. If not, propose what to build instead.
Show the model answer
With zero historical failure examples, the checklist fails on "Historical data exists (or can be constructed)": no failure has ever been recorded: supervised anomaly detection isn't viable yet. The right move is unsupervised/statistical anomaly detection (e.g. an Isolation Forest or simple statistical control limits on sensor readings) to flag unusual readings without labels, paired with a process to have engineers confirm or reject flagged events. That builds the labelled dataset needed for a future supervised model rather than pretending one exists today.
Exercise 2.7 [R]
Read the Wickham piece cited above. Compare its iterative framing to this chapter's three-step sequence (understand the goal → define the ML problem → define success). Where do they agree, and where does iteration break the sequential picture this chapter presents?
Show the model answer
Discussion only. No single answer. Consider: Wickham's iterative framing treats "define the problem" as something that is revisited as data exploration reveals what's measurable. This chapter instead presents the three steps as sequential gates completed before touching data. In practice the two aren't fully in conflict: the problem statement in Procedure 2.1 is meant to be revised, not treated as immutable. But the chapter's ordering is a stronger claim than Wickham's about how much framing must be locked down before EDA begins. Worth considering what breaks if a "final" problem statement is written before any data has been looked at, versus after a first exploratory pass.