Chapter 2
Business Problem Framing
When ML projects fail, it’s not always because the model was bad. They often fail because the model was solving the wrong problem. A customer asks for “AI to improve our operations.” Six months later, a team ships a churn model that scores customers accurately. But operations never plugged it into any workflow. Nothing changed. This chapter is about the translation step that prevents that failure mode: turning a vague business ask into a precisely defined ML problem statement, before anyone writes a line of code.
2.1 Motivation
The gap between “we want AI” and “we have a deployable ML system solving a defined business problem” is where Solutions Architects excel. Every ML project starts with a business problem. Your job is to translate that problem into a precisely defined ML task before touching data. That translation has three steps: understand the business goal, define the ML problem, and define success in measurable terms. These sound obvious. In practice, most projects skip or rush all three. The rest of this chapter shows what happens when they do.
2.2 Exposition
2.2.1 Understanding the Business Goal
The customer’s stated ask is almost never the right ML problem. A few examples:
| Customer says | What they need |
|---|---|
| “We want to predict churn” | Reduce revenue loss → need to identify preventable churn with enough lead time to act |
| “We want to understand our customers better” | Segment customers → unsupervised clustering, not a predictive model |
| “We want a recommendation engine” | Increase conversion → rank items by probability of purchase per user |
| “We want to detect fraud” | Minimise fraud loss → anomaly detection with a specific false-positive budget |
This isn’t a data science problem. It’s a translation problem. Requirements engineering solved a version of it decades before ML existed. The customer’s stated ask is usually a solution, not a need: “we want a recommendation engine” describes a piece of software, not a business outcome. The real requirement sits one level up: increase conversion. The recommendation engine is one way to get there. Maybe not the best one. Digging out the real requirement, not the stated solution, is job zero.
A reliable way to do that: stop talking to just one person. Most ML engagements interview whoever asked for the project and treat that as “the customer’s requirements.” But almost every project has three distinct people or groups behind the ask. They rarely agree on what matters:
- The sponsor pays for the project and owns the business outcome. For a churn model, that’s usually a VP of Customer Success with a revenue target.
- The customer accepts the deliverable and judges whether it solved the problem. That might be a Director of Retention who has to sign off before the model goes live.
- The users are the people who touch the model’s output every day. For a churn model, that’s the retention agents working the accounts the model flags.
Talk to only the sponsor, and the goal statement reads like a budget line: reduce churn, save money. Talk to only the users, and you’ll hear the operational detail no one above them knows. Retention agents often ignore a raw risk score unless it comes with a reason and a suggested next action. A lone number gives them nothing to act on. Miss that, and the model can hit every accuracy target in the spec and still get ignored on the floor. Skipping the users is one of the most common ways an accurate model ships and changes nothing.
Once the goal is understood from all three angles, write it down using three prongs: purpose, advantage, and measurement.
- Purpose: what the project is trying to do, in plain language. Not “build a churn model,” but “identify customers at risk of cancelling with enough lead time for a retention agent to intervene.”
- Advantage: the business benefit, tied to money or a metric leadership already tracks. “Reduce the revenue lost to preventable cancellations.”
- Measurement: how you’ll know it worked, stated as a number. “1,500 prevented cancellations per month” is a measurement. “Improved retention” is not.
Purpose and advantage are usually easy to get agreement on. Measurement is where vague projects get caught: if nobody can state a number, the goal isn’t defined yet, no matter how much everyone nodded along in the kickoff meeting.
This process matters because it determines what gets built. “Predict churn” leads to a classifier. “Reduce preventable revenue loss” leads to a classifier plus a scoring system plus an intervention workflow. The model alone is worthless without the others.
2.2.2 Defining the ML Problem
Once the goal statement exists, with purpose, advantage, and measurement all written down, five questions turn it into something a model can be built against.
What is the label? This is the thing you want to predict. It must be specific, measurable, and tied to a real event. “Customer satisfaction” is not a label. “Customer submits a cancellation request within 30 days” is.
What is the unit of analysis? What does one row in your training data represent? One customer, one transaction, one day per customer? Getting this wrong produces models that leak future information into the past, or that predict at the wrong granularity.
What features are available at prediction time? A model is only useful if you can run it in production. If your most predictive feature requires data that doesn’t exist until after the event you’re predicting, the model is useless. This is one of the most common mistakes in ML projects.
What is the prediction horizon? “Predict churn 30 days before it happens” is a different problem from “predict churn 3 days before.” The former gives time to intervene. The latter not so much.
Is ML the right tool? ML is often unnecessary. Does a simpler rule already work well enough? Is there a pattern to learn, or is the process effectively random? Is there enough data? Appendix A’s ML Suitability Checklist gives a fuller diagnostic.
2.2.3 Defining Success
This is the measurement prong from the goal statement, formalised: a business metric that’s testable, not just a metric that sounds good. “Accuracy” is almost never the right metric in business contexts. If 95% of transactions are legitimate and 5% are fraud, a model that predicts “legitimate” for every transaction scores 95% accuracy and catches zero fraud. Useless. Business success metrics instead connect model outputs to business outcomes:
- Fraud detection: at 1% fraud prevalence, catch 80% of fraud (recall 0.8) while reviewing no more than 2% of transactions manually (precision 0.4 at threshold).
- Churn prevention: score the top 10% of at-risk customers. Intervention converts 15% of those, for a target of 1,500 prevented cancellations per month.
- Recommendation: increase click-through rate from 1.2% to 1.8% in an A/B test ().
Notice none of these are model metrics. They’re business outcomes derived from model performance. A problem statement needs both.
2.2.4 The Precision-Recall Trade-off as a Business Decision
Every classification model produces a probability for each prediction, and a threshold above which you act. Moving the threshold trades precision against recall:
A high, strict threshold flags something only when the model is confident. High precision, low recall, misses real cases. A low, permissive threshold flags anything that looks suspicious. High recall, low precision, many false positives. Which one is right isn’t a data science decision. It’s a business decision: what does a false positive cost, versus a false negative? The business owns that answer, not the model.
There’s a formula version of this trade-off too: the F-beta score, a weighted variant of the familiar F1 score.
sets how much more recall matters than precision. Set and it’s the ordinary F1 score. That treats a false positive and a false negative as equally costly, which is rarely true in a real business. F-beta weights each missed positive times as heavily as each false alarm. So a practical starting point is . A missed fraud case at $200 against a $5 manual review, the same cost pair Procedure 2.4 uses, gives : a score weighted heavily towards recall. This isn’t a substitute for the threshold-cost sweep in Procedure 2.4. That sweep finds the actual optimal threshold, dollar for dollar. The F-beta score is a faster single number for comparing two candidate models before either one has a chosen threshold at all.
2.2.5 The ML Problem Type Taxonomy
| Type | Question being answered | Label type | Example |
|---|---|---|---|
| Binary classification | Will this happen or not? | 0/1, True/False | Churn, fraud, click |
| Multi-class classification | Which category? | One of N classes | Product category, sentiment |
| Regression | How much / when? | Continuous number | Revenue, time-to-event, price |
| Ranking | In what order? | Relative order | Search results, recommendations |
| Clustering | What natural groups exist? | None (unsupervised) | Customer segments, topic groups |
| Anomaly detection | Is this unusual? | None or sparse labels | Fraud, sensor failure, intrusion |
| Time series forecasting | What will happen next? | Future value | Demand, sales, traffic |
Picking the wrong type means building the wrong thing. “Customer segments” sounds like clustering. But if churn labels already exist, supervised classification can find segments that differ in churn rate. That’s far more actionable than an unsupervised grouping.
2.3 Trade-offs and Limits
Wrong framing kills ML projects in three recurring ways. The label problem: a team builds a model to predict “customer engagement score” because that’s the metric on the dashboard. But the actual business goal is subscription renewals. Optimising engagement doesn’t move renewals. Six months of work produces no ROI. The leakage problem: a churn model shows 99% accuracy in testing, then drops to 60% in production. That’s because a feature like “days since last support ticket closed” is only populated after a customer has already churned. The model learned to detect the symptom, not the cause. The horizon problem: a churn model correctly predicts churn 3 days in advance. But the sales team’s intervention process takes 5 days minimum. The model is technically correct and practically useless.
Each of these is a framing failure, not a modelling failure. That’s why it happens before Chapter 11’s classifiers ever enter the picture.
2.4 Procedures
Procedure 2.1 (Write a Structured ML Problem Statement). When to use: at the start of any ML engagement, before any data exploration or modelling. Fill in every field of the template in Appendix A’s Problem Statement Template before proceeding. Leaving a field blank is a signal that the framing isn’t finished.
Procedure 2.2 (Identify the Label). When to use: when the customer describes a goal but hasn’t specified what to predict. Ask, in sequence. What event are we trying to predict? Turn “customer dissatisfaction” into “submits cancellation request.” When does that event occur? This forces a time window, e.g. “within 30 days of billing cycle end.” How do we know it happened? This forces measurability. What’s the unit: per customer, per transaction, per day? Then verify the label exists in the data:
import pandas as pd
df = pd.read_csv('churn_dataset.csv')
print(df['churned_within_30d'].value_counts())
print(df['churned_within_30d'].isnull().sum(), "missing values")
print(f"Positive rate: {df['churned_within_30d'].mean():.1%}")Common mistake: accepting a label the customer has already computed (e.g. an “engagement score” column) without checking whether that computed quantity was the actual business goal.
Procedure 2.3 (Spot When ML Is Overkill). When to use: before committing to build a model. Check whether a simple rule already matches the required performance. Check whether the process is fundamentally random. Check whether there are roughly 1,000 labelled examples per class. A quick baseline makes this concrete. If a hand-written rule already clears the bar, ship the rule instead:
from sklearn.metrics import classification_report
rule_predictions = (
(df['num_support_tickets_90d'] > 2)
& (df['payment_failures_6m'] > 0)
).astype(int)
print(classification_report(
df['churned_within_30d'], rule_predictions
))Procedure 2.4 (Translate Precision vs. Recall into a Business Decision). When to use: whenever a customer asks “how accurate is the model?” Redirect to this framing instead. Attach a dollar cost to a false positive and a false negative. Then sweep thresholds to find the cost-minimising point:
import numpy as np
from sklearn.metrics import confusion_matrix
def business_cost(y_true, y_pred, fp_cost, fn_cost):
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
return fp * fp_cost + fn * fn_cost
# e.g. $5 to manually review, $200 for a missed fraud case
fp_cost, fn_cost = 5, 200
thresholds = np.linspace(0.1, 0.9, 50)
costs = [
business_cost(
y_test, (probs > t).astype(int), fp_cost, fn_cost
)
for t in thresholds
]
optimal_threshold = thresholds[np.argmin(costs)]Rule of thumb: if a false positive is the expensive mistake (e.g. wrongful fraud flag), raise the threshold. If a false negative is the expensive mistake (e.g. missed cancer diagnosis), lower it.
Procedure 2.5 (Check for Data Leakage). When to use: before finalising a feature list. Leakage produces models that look excellent in testing and fail in production. List every candidate feature. Ask whether it could encode information that wouldn’t exist at prediction time. Then sort features by correlation with the label. Anything above roughly 0.9 warrants investigation before use.
2.5 Worked Example
Three Solutions Architect scenarios illustrate how differently “we want AI” cashes out once it’s framed. A telco says: “We’re losing customers and we want to use AI to fix it.” Working through Procedure 2.2 turns this into: predict churned_within_30_days per customer at end of billing cycle, 30 days ahead of the churn date. The model uses account and support-ticket history available at that point. That’s a binary classification problem. A retailer’s “we want to personalise our homepage” becomes a ranking problem over (user, item) pairs with a 200ms latency budget. A manufacturer’s “predict when machines will break down” becomes multi-class classification over failure modes, on hourly-aggregated sensor windows, 48 hours ahead of failure. Three “AI” requests, three structurally different ML problems. None of them obvious from the customer’s original words.
Continuing with the telco scenario, Listing 2.1’s schema is used to generate a synthetic 10,000-row churn dataset. It’s inspected for label distribution before any modelling begins:
feature_cols = [
'months_as_customer', 'monthly_spend',
'num_support_tickets_90d', 'days_since_last_login',
'num_products', 'payment_failures_6m',
]
label_col = 'churned_within_30d'
X = df[feature_cols]
y = df[label_col]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=RANDOM_SEED
)The dataset comes out imbalanced. Far more non-churners than churners in any given month. That’s realistic. Chapter 13 (Cross-Validation & Model Selection) and Chapter 16 (Model Evaluation) return to exactly this imbalance with stratified splits and class weighting. For now, the point is simpler: label separated from features, no leakage, class balance preserved across the split. That is the scaffold every later model in this book is built on.
Procedure 2.4’s threshold sweep needs one thing this chapter hasn’t built yet: a model that outputs a probability. A quick, class-weighted logistic regression on the schema above supplies one, purely to make the mechanics concrete. This isn’t real model-building. Chapter 11 covers that properly.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(class_weight='balanced', random_state=RANDOM_SEED)
clf.fit(X_train, y_train)
probs = clf.predict_proba(X_test)[:, 1]Run Listing 2.3’s sweep against these probabilities, with fp_cost=5 and fn_cost=200:
| Threshold | False positives | False negatives | Cost |
|---|---|---|---|
| 0.3 | 1,869 | 0 | $9,345 |
| 0.5 | 651 | 75 | $18,255 |
| 0.7 | 37 | 123 | $24,785 |
0.3 wins by a wide margin, though it’s an extreme case: this model’s probabilities never drop below about 0.32, so a threshold of 0.3 flags essentially every customer in the test set. It still catches all 131 churners, and at $5 per false positive against $200 per missed churner, flagging everyone costs less than missing anyone. The full 50-point sweep finds the true minimum a little higher, at 0.345 ($9,185). Push the threshold to 0.7 instead, and the model misses 123 of those 131 churners to avoid a small number of false alarms: exactly the mistake a business that loses $200 per missed churner can’t afford. This is Procedure 2.4’s rule of thumb in action. The expensive mistake here is the missed churner, so the threshold belongs low.
import boto3
s3 = boto3.client('s3')
s3.upload_file(
'churn_dataset.csv',
'your-bucket-name',
'datasets/churn/churn_dataset.csv',
)
# Register in SageMaker Feature Store
# (Book 2, Chapter 26 covers this in detail)
# import sagemaker
# feature_store = sagemaker.feature_store.FeatureStore(
# sagemaker_session=session
# )When to prefer SageMaker Feature Store: when multiple teams or models need to share the same feature definitions and require point-in-time correct feature retrieval. That avoids leakage at scale, not just within a single project.
2.6 History and Further Reading
Burkov, Ch. 1–2, gives the formal definition of supervised learning and the anatomy of a dataset (Burkov 2019). Thakur, Ch. 1, takes a practical approach to setting up an ML project from scratch (Thakur 2020). Hadley Wickham’s “The Data Science Process” argues for an iterative framing. It’s worth reading against this chapter’s three-step sequence. See Exercise 2.7 (Wickham, n.d.).
Everything above in “Understanding the Business Goal” (the sponsor/customer/user split, writing a goal as purpose, advantage, and measurement) is adapted from requirements engineering. That’s a discipline with decades of hard-won practice on exactly this translation problem. Suzanne Robertson and James Robertson’s Mastering the Requirements Process is the standard reference. It’s worth buying outright if this translation step interests you more than the modelling that follows it (Robertson and Robertson 2012).
2.7 How This Connects
This chapter builds on Chapter 1, which surveys what kinds of models exist. It enables essentially everything after it. Most directly, Chapter 10 (The Data Science Lifecycle) builds the same framing into a repeatable process. It is often confused with requirements gathering. But framing is not just capturing what the customer wants. It is actively translating that want into a technically precise ML problem statement.
2.8 Exercises
[W] A customer says “we want to predict customer satisfaction.” Explain in one sentence why this is not yet a usable label, and propose a candidate label that would be.
[W] For each of the following, name the correct entry from the Section 2.2.5 taxonomy: (a) grouping customers into segments with no target variable; (b) predicting the dollar value of a customer’s next order; (c) predicting whether a machine will fail in the next 48 hours.
[A] Using the churn dataset schema from Listing 2.4, propose a new candidate feature
days_since_last_price_increase. Using Procedure 2.5, determine whether it is safe to use. Justify your answer.[A] Using Listing 2.5’s probabilities and Listing 2.3’s sweep, verify the total business cost at thresholds 0.3, 0.5, and 0.7 with
fp_cost=5,fn_cost=200. State which threshold minimises cost, and why.[E] The chapter argues accuracy is rarely the right metric. Construct a scenario, distinct from fraud or churn, where accuracy is an acceptable metric. Explain why the class-imbalance argument doesn’t apply there.
[E] A customer asks for “AI to detect anomalies in manufacturing sensor data” but has zero historical examples of past failures. Using the ML Suitability Checklist in Appendix A, decide whether ML is appropriate here. If not, propose what to build instead.
[R] Read the Wickham piece cited above. Compare its iterative framing to this chapter’s three-step sequence (understand the goal → define the ML problem → define success). Where do they agree, and where does iteration break the sequential picture this chapter presents?
2.9 Mental Model
The ML problem statement is a contract. It specifies the prediction target, the unit of analysis, the features available at prediction time, the horizon, and the success metrics. If any one of these is ambiguous, the project fails. Not because of bad algorithms. Because of misaligned expectations. Write the contract before building anything.