Chapter 10
The Data Science Lifecycle
The pre-modelling foundations are now in place: framing a business problem (Chapter 2), exploring and cleaning data (Chapters 7-9). Modelling starts next chapter. It’s time for a change of pace. Knowing the individual skills is not the same as knowing how to run a project. The plan is more useful before the work than after it. Where do you start? When do you stop exploring and start modelling? When is a model “done”? What happens after it ships? Most data science failures are not algorithmic. They are process failures. A model built against the wrong version of the data, three weeks spent tuning before anyone checks whether the baseline was already good enough, a model that ships and silently degrades six months later with nobody watching. This chapter provides a map: the repeatable process that turns a business problem into a monitored, working system, and back into a better business problem.
10.1 Motivation
Every data science project moves through the same broad stages, whatever the use case. Understand the problem, get and understand the data, prepare the data, build a model, evaluate it, deploy it, monitor it, and use what was learned to do it better next time. If you read enough data science texts, you’ll see this drawn as a clean left-to-right pipeline. It’s not. Reality is a loop with paths going backwards at every stage. This chapter provides two things. A formal vocabulary for the stages, so data scientists can talk to each other through established frameworks like CRISP-DM. And a picture of how the process behaves in practice: non-linear, iterative, and full of opportunities for projects to quietly die.
10.2 Exposition
10.2.1 The Stages, Mapped to This Book
| Stage | What happens | Where it lives in this book |
|---|---|---|
| 1. Problem framing | Understand the business goal, translate it into a precise ML problem statement | Chapter 2 |
| 2. Data collection & understanding | Identify data sources, pull data, understand structure, quality, and limitations | Chapters 7-9 |
| 3. Data preparation | Clean, transform, and engineer features into a model-ready form | Chapters 7, 14 |
| 4. Modelling | Select an algorithm family, train candidate models, tune hyperparameters | Chapters 11-20 |
| 5. Evaluation | Measure model performance against the success metric defined in stage 1, not just accuracy, but the business metric | Chapters 13, 16 |
| 6. Deployment | Package the model and serve predictions, batch or real-time | Not in this book: Book 2, Chapter 23 |
| 7. Monitoring | Watch for data drift, model drift, and operational failures in production | Not in this book: Book 2, Chapter 25 |
| 8. Iteration | Feed what was learned, from evaluation and from production, back into a better problem statement | Back to stage 1, Chapter 2 |
This is a compressed version of two well-known industry frameworks. CRISP-DM (Cross-Industry Standard Process for Data Mining) runs through business understanding, data understanding, data preparation, modelling, evaluation, and deployment. Microsoft’s Team Data Science Process (TDSP) adds explicit emphasis on the operational stages (deployment, monitoring, project governance). The structure of this book series deliberately mirrors TDSP’s operational emphasis, because a Solutions Architect is disproportionately valuable precisely at the stages CRISP-DM neglects.
10.2.2 Why the Textbook Diagram Lies
Every lifecycle diagram is drawn as a left-to-right pipeline with a feedback arrow tacked on to the end. That is not how the world works. In a real project, EDA (stage 2) routinely breaks the problem statement (stage 1): the label wanted doesn’t exist, or the class balance makes the original success metric impossible to hit. Back to the drawing board before modelling has even begun. Modelling (stage 4) routinely sends a project back to data preparation (stage 3): a model’s errors reveal a feature bug, or that an important signal isn’t in the data. The two stages cycle dozens of times, rather than being “finished” in sequence. Evaluation (stage 5) routinely reveals that the success metric was wrong. A model hits 0.85 AUC. The team celebrates. Then the business owner explains they care about precision at a specific operating point nobody optimised for. That sends the project back to stage 1’s definition of success. And monitoring (stage 7) is stage 1 for the next iteration: production drift is the start of the next project. Well-run teams treat monitoring alerts as new problem statements rather than tickets to patch.
Reality is a graph. Every stage has an edge back to every earlier stage. Good project management in data science is about recognising which stage you are in, not pretending you are close to some mythical end.
10.2.3 Why Projects Die at Each Stage
| Stage | Common failure mode |
|---|---|
| Problem framing | No one asks “is ML even the right tool?” Months are spent on a problem a SQL query would have solved |
| Data understanding | The team never checks whether labelled data still reflects the current business process |
| Data preparation | Data leakage goes undetected until production, when the leaked feature no longer exists at prediction time |
| Modelling | The team optimises the model metric and never revisits whether it maps to the business metric agreed in stage 1 |
| Evaluation | The model is evaluated once, on one static test set, and never re-evaluated as new data arrives: silent staleness |
| Deployment | The model works in the notebook but nobody built the serving path, so it never reaches production: oddly, this is the most common failure in industry |
| Monitoring | No one owns the model after it’s shipped; drift goes undetected for months |
| Iteration | “Done” is treated as permanent; no process triggers retraining or re-framing |
Deployment is the stage most projects never reach. Industry surveys consistently find that most data science prototypes never make it to production. Which is why deployment, MLOps, and monitoring exist as a dedicated block in Book 2. The modelling techniques in Chapters 11-20 are necessary but not enough. A Solutions Architect’s systems-thinking background is what’s needed: “does it run in a notebook” and “is it a reliable production system” are not the same bar. That distinction is second nature to anyone who has built production systems.
10.3 Trade-offs and Limits
There is a trade-off between rigid process and iteration speed. Formal frameworks like CRISP-DM and TDSP are excellent for communicating with stakeholders and for audits. Followed literally as a waterfall, they slow projects down. Their practical use is as shared vocabulary and checklist, not as a gate that must be cleared before moving to the next stage.
Software engineering had this same argument decades earlier. Winston Royce’s 1970 paper is usually credited as the origin of “waterfall,” but Royce never used that word. His own paper calls the strict sequential version risky, and proposes feedback loops back to earlier phases as a required part of the process, not a sign something went wrong. The rigid, one-way reading came later, from people summarising the paper rather than reading it closely. The Agile Manifesto (2001) formalised the fix decades on: short iterative cycles instead of long sequential phases, built specifically to solve the waterfall problem software teams had been living with since Royce’s warning went unheeded. CRISP-DM and TDSP inherit the identical tension: a diagram drawn as a sequence, describing a process that was always meant to loop back.
There is a matching trade-off between depth of upfront framing and speed to first result. Spending three weeks perfecting a problem statement before touching data is its own failure mode. Nobody can know what’s realistic until the data has been seen. The practical approach is a lightweight problem statement (Chapter 2’s template, filled in quickly), followed by a fast pass through EDA, followed by a revised problem statement. Iterate the contract. Don’t treat the draft as final.
Robertson and Robertson’s Mastering the Requirements Process, already the standard reference cited in Chapter 2, gives this trade-off a name (Robertson and Robertson 2012). Their Formality Guide sorts projects into Rabbit, Horse, and Elephant, with Horse as the default middle case. A Rabbit project is small, fast, and short-lived: it discovers requirements in small units, implements a small increment, and uses that working increment to get feedback from stakeholders, rather than writing a complete specification first. An Elephant project is the opposite: long duration, many distributed stakeholders, and a genuine need for a complete, formal specification before work starts. Data science projects are Rabbits, or should be treated as such whenever possible. The failure mode this section describes is treating one like an Elephant: writing the specification an Elephant needs for a project the data will reshape within a week.
10.4 Procedures
Procedure 10.1 (Write a Project Charter). When to use: at project kick-off, before any data work: a lightweight companion to Chapter 2’s problem statement template, scoped to the whole project rather than the ML problem alone.
## Project Charter
**Project name:** [Name]
**Sponsor / business owner:** [Who owns the outcome and
signs off on "done"]
**Lifecycle stage owners:**
- Data: [who owns data access/quality]
- Modelling: [who builds and evaluates models]
- Deployment: [who owns the serving path]
- Monitoring: [who gets paged when it breaks -- must
be a named person, not "the team"]
**Problem statement:** [link to the Chapter 2 ML problem
statement]
**Timeline:**
- Data understanding complete by: [date]
- First baseline model by: [date]
- Go/no-go decision by: [date]
- Production deployment by: [date]
**Definition of done:** [Not "model built" -- "model
deployed, monitored, and business metric moving"]
**Kill criteria:** [Under what conditions does this project
get cancelled? e.g. "baseline model doesn't beat the
existing rule by 10% relative"]Common mistake: skipping “kill criteria.” Projects without an explicit exit condition tend to survive purely on sunk cost, consuming months of effort on a problem that was never going to work.
Procedure 10.2 (Run a Pre-Mortem). When to use: right after the charter is written, before data work begins: a 30-minute exercise that surfaces risks while they are still cheap to fix.
## Pre-Mortem: Assume this project failed.
## It's 6 months from now. Why?
1. Data reasons it failed:
- [ ] The label wasn't recorded reliably
- [ ] Class imbalance was worse than expected
- [ ] A key feature isn't available at prediction time
(leakage discovered late)
- [ ] Data access/pipeline broke and nobody noticed for weeks
2. Modelling reasons it failed:
- [ ] The offline metric didn't translate to a business outcome
- [ ] The model needed retraining more often than
anyone budgeted for
3. Organisational reasons it failed:
- [ ] Nobody owned deployment, so it stayed in a notebook
- [ ] Nobody owned monitoring, so drift went undetected
- [ ] The business owner changed and the new owner didn't buy in
4. For every checked box: what's the mitigation, and who owns it?Common mistake: treating the pre-mortem as a formality and skipping straight to “no risks, let’s build.” The exercise only works if the team tries to find the failure mode before it happens.
Procedure 10.3 (Decide “Is This Even a Project?”). When to use: when a stakeholder brings an ask and it needs quick triage: full lifecycle project, or a five-minute answer?
def triage(ask: dict) -> str:
"""Quick triage for an incoming DS/ML request."""
if ask.get("simple_rule_suffices"):
return (
"Not a project -- write the rule, ship it "
"in a day (Chapter 2)"
)
if ask.get("one_off_analysis_only"):
return (
"Not a lifecycle project -- this is an "
"analytics/EDA task (Chapter 9)"
)
if not ask.get("historical_labelled_data_exists"):
return (
"Not ready -- this is a data collection "
"project first, an ML project second"
)
return "Full lifecycle project -- use the charter template"
ask = {
"simple_rule_suffices": False,
"one_off_analysis_only": False,
"historical_labelled_data_exists": True,
}
print(triage(ask))Procedure 10.4 (Structure a Data Science Project Repository). When to use: at the start of every project: a consistent layout makes handoffs and audits far easier.
project_name/
|-- README.md # Charter, kept up to date
|-- data/
| |-- raw/ # Never modified -- read-only
| |-- interim/ # Intermediate cleaning outputs
| `-- processed/ # Model-ready datasets
|-- notebooks/
| |-- 01_eda.ipynb
| |-- 02_feature_engineering.ipynb
| `-- 03_modelling.ipynb
|-- src/
| |-- data_prep.py # Reusable, tested code
| |-- features.py
| |-- train.py
| `-- evaluate.py
|-- models/ # Serialised artefacts, versioned
|-- monitoring/
| `-- drift_checks.py
|-- tests/
`-- requirements.txtCommon mistake: leaving all logic in notebooks. Notebooks are for exploration (stages 2-4); anything that needs to run in production (data prep, feature engineering, inference) belongs in tested .py modules that the notebooks call, not duplicate.
Procedure 10.5 (Run a Project Retrospective). When to use: at the end of every lifecycle iteration, whether the project shipped, was killed, or is looping back to stage 1.
## Retrospective: [Project Name] -- [Date]
**What we set out to do:** [Original problem statement]
**What actually happened:** [Outcome -- shipped / killed /
iterating]
**What went well:**
- [...]
**What we'd do differently:**
- [...]
**Where did we loop back, and why?**
- [Stage] -> [Earlier stage]: [reason]
**Is there a next iteration? If so, what's the revised
problem statement?**
- [...]
**Action items for the team's process (not just this project):**
- [...]Common mistake: only running retrospectives on failed projects. Successful projects have just as much to teach about which stage transitions worked smoothly and why. Capture that too.
10.5 Worked Example
The tutorial notebook carries a single churn-prediction problem, the same telco scenario from Chapter 2, through all eight lifecycle stages in miniature, in one notebook. It is worth telling because the result is a project that does not clear its own bar. That’s the realistic outcome of a first pass, not a flaw in the exercise.
Stage 1 restates the abbreviated problem statement from Chapter 2: predict churned_within_30d, one row per customer at end of billing cycle. Success is defined as precision 0.40 at recall 0.60 against the model metric, and beating the existing simple rule against the business metric. Stage 2 generates a synthetic population of 8,000 customers with a churn-generating process built from support tickets, payment failures, login recency, tenure, plan type, and product count. The resulting churn rate comes out to 7.0%. That’s realistically imbalanced, and low enough that accuracy alone would be almost meaningless as a metric, exactly the trap Chapter 2 warns against.
Stage 3 splits the data (80/20, stratified, random_state=42) before touching anything further: 6,400 training rows and 1,600 test rows, each preserving the 7% churn rate. It fits a StandardScaler on the training fold only, to avoid leaking test-set statistics into the features. Stage 4 then insists on two baselines before any real model is trained:
dummy = DummyClassifier(
strategy='most_frequent', random_state=RANDOM_SEED
)
dummy.fit(X_train, y_train)
# Looser than Chapter 2's rule (> 2 tickets).
# X_test is unscaled: the scaler writes X_test_scaled.
rule_preds = (
(X_test['num_support_tickets_90d'] >= 2)
& (X_test['payment_failures_6m'] >= 1)
).astype(int)On the actual generated data, the naive “always predict no churn” baseline scores 93% accuracy while catching zero churners (precision and recall both 0.00 on the positive class). That’s a direct example of why accuracy is the wrong metric here. The hand-written rule does slightly better on recall but is still weak: 9% precision, 8% recall. Only then does Stage 4 train a logistic regression with class weighting, which reaches AUC-ROC 0.638, a real but modest improvement over guessing.
Stage 5 is where the lifecycle’s honesty pays off. Sweeping thresholds for the point that clears recall 0.60 finds a threshold of 0.493. But the precision achieved there is only 0.103, well short of the 0.40 target set in Stage 1. At that threshold the model flags 661 of 1,600 test customers, more than six times as many as the simple rule’s 100, for a precision barely above the rule’s. The model does not meet the business contract. This is exactly the “loop-back” moment the chapter’s Exposition describes in the abstract, made concrete. With only six raw features, the right move is not to declare victory on a good-looking AUC. It’s to return to Stage 3 for better features (Chapter 14’s feature engineering), or to Stage 1 to renegotiate the metric, not to proceed to deployment.
The notebook proceeds through Stages 6-8 anyway, explicitly labelled as showing what those stages look like, rather than a validated next step for this particular model. A deployment plan favours nightly batch scoring, since the intervention workflow takes hours, so real-time serving doesn’t help. A monitoring plan watches feature drift, predicted-churn-rate drift, and realised precision on a rolling window. And a retrospective records the loop-back: Stage 5 to Stage 3 for better features, and potentially Stage 5 to Stage 1 if better features still don’t clear the bar. The retrospective is a Python dictionary in the notebook, not a prose afterthought. That matters: writing the loop-back down in a structured form, easy to find again later, is what makes the next iteration happen instead of quietly dying. It echoes Procedure 10.1’s kill criteria and named owners.
# SageMaker Projects gives a templated, governed version of
# the repo structure above, with CI/CD wiring for the
# deployment and monitoring stages built in.
import sagemaker
# Projects are typically created via the SageMaker Studio UI
# or Service Catalog, using a pre-built or custom MLOps
# template (build/train/deploy pipeline).
# Once created, each project provisions:
# - A Git repo (CodeCommit, or GitHub/GitLab/Bitbucket
# through a connection) mirroring the structure above
# - CodePipeline stages for build -> train -> deploy
# - A Model Registry entry per trained model version
# For a lighter-weight equivalent without full SageMaker
# Projects overhead, a SageMaker Pipeline captures the
# stages directly in code:
# from sagemaker.workflow.pipeline import Pipeline
# pipeline = Pipeline(
# name="churn-lifecycle-pipeline",
# # mirrors stages 3-6
# steps=[
# data_prep_step, train_step, eval_step, register_step
# ],
# )When to prefer SageMaker Projects: once a project has moved from “one person’s notebook” to “a system with a defined deployment and monitoring owner”, i.e., once it has cleared its own pre-mortem and is heading to production. Don’t provision full MLOps scaffolding for stage-1/2 exploratory work.
10.6 History and Further Reading
The original CRISP-DM paper by Wirth and Hipp remains the most widely cited process framework in the field. It’s worth reading in full, even though it predates the operational emphasis this chapter argues for. Microsoft’s public documentation on the Team Data Science Process is the closest published equivalent to this chapter’s own stage structure, particularly its explicit treatment of deployment and monitoring as first-class stages rather than afterthoughts. Thakur’s opening chapter gives a practitioner’s ground-level view of what a data science project looks like day to day, complementing this chapter’s more structural framing (Thakur 2020). Burkov’s first chapter, read alongside this one, gives the formal anatomy of a supervised learning project that this chapter’s Stage 3-5 sequence builds directly on (Burkov 2019, 1–15).
10.7 How This Connects
This chapter builds on Chapter 2, which is where the lifecycle starts and which gets revisited every time the lifecycle loops back. It also builds on Chapter 9, whose EDA structure fills stage 2. It enables Book 2’s chapters on experiment tracking, deployment, MLOps, and monitoring, pointing ahead to why each of those exists and where it fits before any of them are covered in depth. It is often confused with a project plan or Gantt chart. But the lifecycle is not a schedule. It is a project state machine, detailing what states a project can be in and how it moves between them. Two projects following the exact same lifecycle can take a week or a year, depending on how many loops back they need.
10.8 Exercises
[W] Name the eight lifecycle stages in order. For each, name one chapter (or block of chapters) in this book where that stage’s skills live.
[W] Explain in two or three sentences why the textbook left-to-right pipeline diagram is misleading, using at least one concrete example of a backward loop from this chapter.
[A] Using Listing 10.3, construct an
askdictionary describing a hypothetical stakeholder request wherehistorical_labelled_data_existsisFalse. Report what the function returns, and why that’s the correct triage outcome.[A] Using the logic of Listing 10.6 as a model, compute what accuracy the naive “predict majority class” baseline would score on a dataset with a 3% positive rate. Explain why that number is a poor argument for using the naive baseline in production.
[A] The tutorial’s logistic regression achieved AUC-ROC of approximately 0.638 but failed to meet the precision/recall target from Stage 1. Using this chapter’s Stage 5 discussion, name the two specific stages the project should loop back to. For each, give one concrete change that might close the gap.
[E] Rewrite the tutorial’s Stage 6 deployment plan for a scenario where the business owner instead wants real-time scoring at the moment a customer contacts support, rather than nightly batch scoring. What changes about the serving pattern, the latency budget, and the rollback plan?
[E] A team tells you they “follow CRISP-DM,” but their retrospectives show every project has shipped a model that was never revisited after deployment. Using Appendix A’s Framework Comparison, explain which stage CRISP-DM under-specifies that likely explains this pattern. Explain what TDSP would have required instead.
[R] Read the original CRISP-DM paper (Wirth & Hipp, 2000) or the Microsoft TDSP documentation. Compare its treatment of “iteration” to this chapter’s Stage 8. Does the source treat iteration as a distinct, named stage with its own deliverable, or as an implicit property of the whole process? What does that difference suggest about how seriously each framework takes the “loop back” problem?
10.9 Mental Model
The data science lifecycle is not a straight road walked once. It is a loop walked many times. The skill is knowing which stage you are in now versus which stage you think you are in. If a model’s metrics look great but nobody agreed on what “great” means for the business, the project is still in stage 1. No matter how much modelling code has been written. If a model is in production but no one is watching it, the project is not done. It just hasn’t discovered it’s back at the beginning.