◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Appendix A

Quick Reference

Scan, don’t read. This appendix collects each chapter’s lookup material, organised by chapter: key terms, API summaries, formulas, decision guides, and metrics. Keep it open while working; it is not meant to be read start to finish.

Chapter 1 — History & Landscape

AI / ML Timeline at a Glance

Year Event Why it matters
1950 Turing Test An early operational test for machine intelligence
1956 Dartmouth workshop Founds AI as a field (name coined in the 1955 proposal)
1986 Backpropagation popularised Makes multi-layer networks trainable
1995 SVMs Strong theoretical foundations for classification
1997 LSTM Recurrent networks that learn long-range dependencies
2001 Random Forests Still a strong tabular baseline 20+ years later
2009 ImageNet released Large labelled dataset enables scale
2012 AlexNet Deep learning goes mainstream
2017 Transformer (“Attention Is All You Need”) Foundation of modern LLMs
2018 BERT Transfer learning for NLP
2020 GPT-3 Few-shot learning; scale changes the game
2022 ChatGPT RLHF + instruction tuning → conversational AI

The Three Eras

Era Rough dates Core idea Still used?
Rules & Expert Systems 1950s-1980s Hand-coded logic Rarely, in very narrow domains
Statistical ML 1980s-2010s Learn patterns from labelled data Yes — dominates tabular ML
Deep Learning / Foundation Models 2012-now Scale neural nets on massive data Yes — dominates unstructured data

Role Definitions

Role Primary tool Spends most time on
Data Analyst SQL, BI tools Describing what happened
Data Engineer Spark, dbt, Kafka Moving and storing data
Data Scientist sklearn, PyTorch Building predictive models
ML Engineer MLflow, Docker, k8s Making models work in production
Research Scientist Papers, JAX/PyTorch Advancing the state of the art
AI/ML Solutions Architect All of the above Designing systems, advising customers

Nested Scope of Terms

Artificial Intelligence
`-- Machine Learning
    |-- Classical ML (linear models, trees, SVMs, ensembles)
    `-- Deep Learning
        |-- CNNs (images)
        |-- RNNs / LSTMs (sequences)
        `-- Transformers (text, code, images, multimodal)
            `-- Foundation Models (pretrained, then adapted;
                |                      text, vision, multimodal)
                `-- Large Language Models (GPT, Claude, Llama, Gemini)

When to Use Which Approach

Situation Reach for
Tabular data, clear features, <1M rows XGBoost / LightGBM / Random Forest
Tabular data, need to explain decisions Logistic Regression + SHAP
Images, audio, video CNN or Vision Transformer
Text: classification, extraction Fine-tuned BERT or LLM
Text: generation, chat, summarisation Foundation model (prompt or fine-tune)
Time series forecasting Prophet / ARIMA / Temporal Fusion Transformer
Small dataset, simple relationship Linear / logistic regression
Need to deploy cheaply Classical ML almost always wins

Essential pandas One-Liners

df.shape                          # (rows, cols)
df.dtypes                         # column types
df.isnull().sum()                 # missing values per column
df.describe()                     # summary stats (numeric)
df['col'].value_counts()          # frequency of each category
df.groupby('col')['val'].mean()   # grouped mean
df.sort_values('col', ascending=False)  # sort descending
df[df['col'] > threshold]         # filter rows

Essential matplotlib One-Liners

plt.figure(figsize=(12, 5))             # set figure size
plt.plot(x, y, label='name', color='#4E79A7')   # line
plt.scatter(x, y, s=50, alpha=0.5)     # scatter
plt.axvline(x=val, linestyle='--')     # vertical reference line
plt.xlabel('X Label')
plt.ylabel('Y Label')
plt.title('Title')
plt.legend()
plt.tight_layout()
plt.savefig('file.png', dpi=150, bbox_inches='tight')
plt.show()

Chapter 2 — Business Problem Framing

Key Terms

Term Definition
Label The exact, measurable quantity being predicted, tied to a real event
Unit of analysis What one row of training data represents
Prediction horizon How far in advance of the event the prediction must be made
Data leakage A feature that encodes information unavailable at prediction time

ML Problem Type Taxonomy

Type Label Algorithm family Use when
Binary classification 0 or 1 Logistic regression, tree-based, neural net Predict yes/no event
Multi-class classification One of N classes Same as binary + softmax Predict one category from several
Multi-label classification Multiple 0/1 flags Sigmoid per class One item can belong to multiple categories
Regression Continuous number Linear regression, tree-based, neural net Predict a quantity
Ranking Relative order LTR models, collaborative filtering Order items by relevance per user
Clustering None (unsupervised) K-means, DBSCAN, hierarchical Discover natural groups
Anomaly detection Rare or unlabelled Isolation Forest, Autoencoder, statistical Find unusual examples
Time series forecasting Future value ARIMA, Prophet, TFT Predict next N values in a sequence
Sequence labelling Tag per token CRF, LSTM, BERT NER, POS tagging

Problem Statement Template

GOAL STATEMENT
  Purpose:      what this project is trying to do, in plain language
  Advantage:    the business benefit, tied to money or a tracked metric
  Measurement:  how success is verified, stated as a number

ML PROBLEM DEFINITION
  Label:               the exact, measurable event being predicted
  Unit of analysis:    what one training row represents
  Features available:  confirmed available at prediction time, not after
  Prediction horizon:  how far ahead of the event the prediction is made
  Is ML the right tool?  checked against the Suitability Checklist below

SUCCESS METRIC
  Business outcome:  ________________________
  Target number:     ________________________

A blank field is the signal Procedure 2.1 warns about: the framing isn’t finished yet.

The ML Suitability Checklist

[ ] There is a clear, measurable prediction target
[ ] Historical data exists (or can be constructed)
[ ] Features/label relationship is stable over time
[ ] Enough examples exist (rule of thumb: >= 1,000/class)
[ ] Features at prediction time are enough to predict
[ ] A simple rule does NOT already work well enough
[ ] The cost-benefit of ML vs. rule justifies the effort
[ ] The action enabled by the prediction is operationalised

AWS SageMaker Quick Reference

Task SageMaker approach Documentation
Share a structured dataset across a team Upload to S3, register in SageMaker Feature Store sagemaker.feature_store

Chapter 3 — Maths Foundations: Statistics, Probability & Linear Algebra

Key Terms

Term Definition
Prior P(A)P(A): belief about a hypothesis before seeing evidence
Likelihood P(B∣A)P(B \mid A): probability of the evidence given the hypothesis
Posterior P(A∣B)P(A \mid B): updated belief after accounting for evidence
Variance Average squared deviation from the mean, σ2=1n∑i(xi−μ)2\sigma^2 = \frac{1}{n}\sum_i (x_i-\mu)^2
Correlation (Pearson r) Scale-independent measure of linear association, r=Cov⁡(X,Y)/(σXσY)∈[−1,1]r = \operatorname{Cov}(X,Y)/(\sigma_X\sigma_Y) \in [-1,1]
p-value Probability, under H0H_0, of data at least as extreme as what was observed
Effect size A measure (e.g. Cohen’s dd) of how large a difference is, independent of sample size
Normal equations The closed-form least-squares solution w=(X⊤X)−1X⊤yw = (X^\top X)^{-1}X^\top y

Probability Distributions Cheat Sheet

Distribution Parameters Mean Variance scipy / numpy Use when
Gaussian μ,σ\mu, \sigma μ\mu σ2\sigma^2 stats.norm(mu, sigma) Continuous, symmetric quantities
Bernoulli pp pp p(1−p)p(1-p) stats.bernoulli(p) Single binary outcome
Binomial n,pn, p npnp np(1−p)np(1-p) stats.binom(n, p) Count of successes in nn trials
Poisson λ\lambda λ\lambda λ\lambda stats.poisson(lam) Count of rare events in a fixed interval
Uniform a,ba, b (a+b)/2(a+b)/2 (b−a)2/12(b-a)^2/12 np.random.uniform(a,b) Equal probability over a range
Exponential λ\lambda 1/λ1/\lambda 1/λ21/\lambda^2 stats.expon(scale=1/lam) Time between events
Beta α,β\alpha, \beta α/(α+β)\alpha/(\alpha+\beta) αβ(α+β)2(α+β+1)\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)} stats.beta(a, b) Probability of a probability (Bayesian)

Bayes’ Theorem

P(A∣B)=P(B∣A) P(A)P(B),P(B)=∑iP(B∣Ai) P(Ai)P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}, \qquad P(B) = \sum_i P(B \mid A_i)\,P(A_i)

In ML: BB = observed features (evidence), AA = the label (hypothesis). P(A)P(A) is the prior (base rate); P(B∣A)P(B\mid A) is the likelihood; P(A∣B)P(A \mid B) is the posterior: the quantity a classifier is ultimately trying to produce.

Descriptive Statistics Summary

Statistic Formula numpy Notes
Mean xˉ=∑xi/n\bar{x} = \sum x_i / n np.mean(x) Sensitive to outliers
Median Middle value np.median(x) Robust to outliers
Variance σ2=∑(xi−μ)2/n\sigma^2 = \sum(x_i-\mu)^2/n np.var(x) Use ddof=1 for sample variance
Std dev σ=σ2\sigma = \sqrt{\sigma^2} np.std(x) Same units as xx
Covariance ∑(xi−μx)(yi−μy)/n\sum(x_i-\mu_x)(y_i-\mu_y)/n np.cov(x,y) Not normalised; scale-dependent. np.cov divides by n−1n-1 by default
Correlation Cov⁡(X,Y)\operatorname{Cov}(X,Y) / (σXσY)(\sigma_X\sigma_Y) np.corrcoef(x,y) Scale-independent, [−1,1][-1,1]
IQR Q3−Q1Q_3 - Q_1 np.percentile(x,[25,75]) Robust spread measure

Common Statistical Tests

Test Use when Assumes scipy call
Shapiro-Wilk Test if data is Gaussian (n<5000n<5000) — stats.shapiro(x)
t-test (independent) Compare means of 2 groups Gaussian, equal variance stats.ttest_ind(a, b)
Welch’s t-test Compare means of 2 groups (unequal variance) Gaussian stats.ttest_ind(a, b, equal_var=False)
Mann-Whitney U Compare 2 groups (non-parametric) None stats.mannwhitneyu(a, b)
ANOVA Compare means of 3+ groups Gaussian, equal variance stats.f_oneway(*groups)
Chi-squared Association between 2 categorical variables Expected counts >5>5 stats.chi2_contingency(table)
Pearson r Linear correlation Linear relationship; Gaussian only for the p-value stats.pearsonr(x, y)
Spearman ρ\rho Monotonic correlation (non-parametric) None stats.spearmanr(x, y)

Linear Algebra Operations

Operation Notation numpy Shape
Vector dot product a⋅b\mathbf{a}\cdot\mathbf{b} a @ b (n,)⋅(n,)→(n,)\cdot(n,) \to scalar
Matrix multiply ABAB A @ B (m,n)@(n,k)→(m,k)(m,n)@(n,k) \to (m,k)
Transpose A⊤A^\top A.T (m,n)→(n,m)(m,n) \to (n,m)
Inverse A−1A^{-1} np.linalg.inv(A) (n,n)→(n,n)(n,n) \to (n,n)
Solve Ax=bAx=b x=A−1bx = A^{-1}b np.linalg.solve(A, b) More stable than inv
Eigendecomposition Av=λvAv = \lambda v np.linalg.eig(A) Returns (eigenvalues, eigenvectors)
SVD A=UΣV⊤A = U\Sigma V^\top np.linalg.svd(A) Returns (U, s, Vt)
L2 norm ∥v∥2\lVert v\rVert_2 np.linalg.norm(v) → scalar
L1 norm ∥v∥1\lVert v\rVert_1 np.linalg.norm(v, ord=1) → scalar
Element-wise multiply A⊙BA \odot B A * B Same shape as A,BA, B

Normal Equations for Linear Regression

y^=Xw,w=(X⊤X)−1X⊤y\hat{y} = Xw, \qquad w = (X^\top X)^{-1}X^\top y

X_bias = np.column_stack([np.ones(len(X)), X])
w = np.linalg.solve(X_bias.T @ X_bias, X_bias.T @ y)
# w[0] = intercept, w[1:] = feature weights

Limitations: O(M3)O(M^3) complexity in the number of features MM, slow for large MM. Fails if X⊤XX^\top X is singular (collinear features). For collinear features, use Ridge (Chapter 18) or a least-squares solver such as np.linalg.lstsq. For very large MM, use gradient descent (Chapter 4).

Correlation Interpretation Guide

∣r∣\lvert r\rvert range Strength
0.9 – 1.0 Very strong
0.7 – 0.9 Strong
0.5 – 0.7 Moderate
0.3 – 0.5 Weak
0.0 – 0.3 Negligible

Caveats: Pearson rr measures linear correlation only; r=0r=0 does not imply independence (could be non-linear); correlation is not causation; Spearman ρ\rho is more robust to outliers and handles monotonic non-linear relationships.

AWS SageMaker Quick Reference

Task SageMaker approach Documentation
Statistical analysis via UI, no code SageMaker Data Wrangler sagemaker.wrangler
Statistics on datasets too large for local memory (roughly >10>10 GB) SageMaker Processing Jobs (ScriptProcessor) sagemaker.processing.ScriptProcessor

Chapter 4 — Maths Foundations: Calculus & Optimisation

Key Terms

Term Definition
Derivative Rate of change of a function at a point, dfdx\frac{df}{dx}
Gradient Vector of partial derivatives, ∇f(w)\nabla f(\mathbf{w}); points in the direction of steepest ascent
Learning rate (α\alpha) Step size in the gradient descent update rule
Backpropagation Repeated application of the chain rule to compute gradients through a composed (layered) function
Convex loss A loss surface where every local minimum is a global minimum; gradient descent reaches it with a small enough learning rate
Hessian Matrix of second derivatives; used by second-order optimisers, M×MM \times M for MM parameters

Gradient Descent Update Rule

w←w−α ∇L(w)\mathbf{w} \leftarrow \mathbf{w} - \alpha \, \nabla L(\mathbf{w})

Gradient Descent Variants

Variant Batch size numpy pattern Pros Cons
Batch GD All NN examples X.T @ (X@w - y) Exact gradient Slow for large NN
Stochastic GD (SGD) 1 example X[i] @ (X[i]@w - y[i]) Fast; noise can escape local minima Very noisy, hard to converge
Mini-batch GD BB examples (32-256) X[batch].T @ (X[batch]@w - y[batch]) Best of both; standard in practice Requires batch-size tuning

Common Loss Functions

Loss Formula Use for Gradient
MSE 1N∑(y^−y)2\frac{1}{N}\sum(\hat{y}-y)^2 Regression 2NX⊤(Xw−y)\frac{2}{N}X^\top(X\mathbf{w}-y)
MAE 1N∑∣y^−y∣\frac{1}{N}\sum|\hat{y}-y| Regression, outlier-robust 1NX⊤sign⁡(Xw−y)\frac{1}{N}X^\top \operatorname{sign}(X\mathbf{w}-y)
Huber MSE if ∣e∣<δ|e|<\delta, else MAE Regression, robust Hybrid of MSE/MAE gradients
Binary cross-entropy −[ylog⁡y^+(1−y)log⁡(1−y^)]-\big[y\log\hat{y}+(1-y)\log(1-\hat{y})\big] Binary classification (y^−y)(\hat{y}-y) with respect to the logit, with sigmoid output
Categorical cross-entropy −∑kyklog⁡y^k-\sum_k y_k \log \hat{y}_k Multi-class classification (y^−y)(\hat{y}-y) with respect to the logits, with softmax output
Hinge max⁡(0,1−yy^)\max(0, 1-y\hat{y}) SVM −y-y if yy^<1y\hat{y}<1, else 00

Optimiser Comparison

Optimiser Update rule Learning-rate sensitivity Best for
SGD w←w−α∇L\mathbf{w} \leftarrow \mathbf{w}-\alpha\nabla L High Simple models, full manual control
SGD + momentum v←βv+(1−β)∇L\mathbf{v}\leftarrow\beta\mathbf{v}+(1-\beta)\nabla L; w←w−αv\mathbf{w}\leftarrow\mathbf{w}-\alpha\mathbf{v} Moderate Deep learning with careful tuning
Adam Adaptive per-parameter learning rate Low Default for neural networks and fine-tuning
AdamW Adam + decoupled weight decay Low Transformer training (standard)
L-BFGS Quasi-Newton (2nd-order) Very low Smooth, full-batch problems (classical ML); memory grows linearly with parameter count
RMSprop Adaptive learning rate, no momentum Moderate RNNs, non-stationary objectives

Adam Algorithm (Simplified)

alpha, beta1, beta2, eps = 0.001, 0.9, 0.999, 1e-8
m, v = np.zeros_like(w), np.zeros_like(w)

for t in range(1, n_steps + 1):
    g = gradient(w)
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * g**2
    m_hat = m / (1 - beta1**t)
    v_hat = v / (1 - beta2**t)
    w = w - alpha * m_hat / (np.sqrt(v_hat) + eps)

Learning Rate Schedule Types

Schedule Behaviour When to use
Constant Fixed throughout training Simple baseline
Step decay Halve every NN epochs Traditional CNN training
Cosine decay Smooth decay to 0 General neural network training
Warm-up + cosine Linear ramp then cosine decay Transformer training (standard)
Reduce on plateau Reduce when validation loss stalls When unsure of a fixed schedule
Cyclic Oscillates between min/max Helps explore the loss landscape

αt=α0⋅12(1+cos⁡(πtT)),t=current epoch, T=total epochs\alpha_t = \alpha_0 \cdot \tfrac{1}{2}\left(1 + \cos\left(\frac{\pi t}{T}\right)\right), \qquad t = \text{current epoch},\ T = \text{total epochs}

Key Calculus Facts for ML

Concept Formula ML context
Derivative of x2x^2 2x2x MSE gradient
Derivative of log⁡(x)\log(x) 1/x1/x Cross-entropy gradient
Derivative of σ(x)\sigma(x) σ(x)(1−σ(x))\sigma(x)(1-\sigma(x)) Sigmoid layer backprop
Derivative of relu⁡(x)\operatorname{relu}(x) 11 if x>0x>0 else 00 ReLU layer backprop
Chain rule ddxf(g(x))=f′(g(x)) g′(x)\frac{d}{dx}f(g(x)) = f'(g(x))\,g'(x) Backpropagation
Product rule ddx[f⋅g]=f′g+fg′\frac{d}{dx}[f \cdot g] = f'g + fg' Occasionally needed

scipy.optimize Cheat Sheet

import scipy.optimize as opt

# Minimise scalar fn
result = opt.minimize(fun, x0, method='L-BFGS-B', jac=gradient_fn)
# Find f(x) = 0
result = opt.root(fun, x0)
# 1D, no gradient needed
result = opt.minimize_scalar(fun, bounds=(a, b), method='bounded')
# Least squares fit
popt, pcov = opt.curve_fit(
    model_fn, xdata, ydata, p0=initial_params
)

result.x        # optimal parameters
result.fun      # final function value (loss)
result.success  # bool: did it converge?
result.nit      # number of iterations
Method Best for Notes
'L-BFGS-B' General smooth optimisation Usually best default
'Nelder-Mead' No gradient available Slow but robust
'SLSQP' Constrained optimisation Handles inequality constraints
'Newton-CG' Smooth, exact gradient available Fast convergence

Diagnosing Training Problems

Symptom Likely cause Fix
Loss goes to NaN LR too high, or numerical overflow Reduce LR 10x; add gradient clipping
Loss oscillates LR too high Reduce LR 3-10x
Loss plateaus early LR too low, or converged to local min Increase LR; use momentum/Adam
Val loss increases while train decreases Overfitting Regularisation, more data
Both train and val loss plateau high Underfitting Larger model, more epochs, better features
Loss decreases but very slowly LR too low, or vanishing gradients LR schedule; batch norm; residual connections

AWS SageMaker Quick Reference

Task SageMaker approach Documentation
Set training hyperparameters (learning rate, epochs, batch size) sagemaker.estimator.Estimator(hyperparameters={...}) sagemaker.estimator
Search over learning rate / hyperparameters automatically SageMaker Automatic Model Tuning (Bayesian optimisation) Covered in a later chapter on hyperparameter tuning

Chapter 5 — Information Theory Basics

Key Terms

Term Definition
Self-information I(p)=−log⁡(p)I(p) = -\log(p): surprise of a single event with probability pp
Entropy H(P)H(P) Average information content of a distribution; measures uncertainty
Cross-entropy H(P,Q)H(P, Q) Average cost of using predicted distribution QQ when the truth is PP
KL divergence KL(P∥Q)\mathrm{KL}(P\|Q) Directed “distance” of QQ from PP; always ≥0\geq 0, asymmetric
Mutual information I(X;Y)I(X; Y) How much knowing XX reduces uncertainty about YY (and vice versa); zero iff independent
Nat vs. bit Unit of information: natural log gives nats, log⁡2\log_2 gives bits; 1 nat≈1.4427 bits1\text{ nat} \approx 1.4427\text{ bits}

Key Formulas

Name Formula When you use it
Self-information I(p)=−log⁡(p)I(p) = -\log(p) Building block for everything below
Entropy H(P)=−∑xP(x)log⁡P(x)H(P) = -\sum_x P(x) \log P(x) Node impurity (trees), dataset uncertainty
Cross-entropy H(P,Q)=−∑xP(x)log⁡Q(x)H(P, Q) = -\sum_x P(x) \log Q(x) Classification loss function
Binary cross-entropy L=−[ylog⁡(y^)+(1−y)log⁡(1−y^)]L = -[y \log(\hat y) + (1-y) \log(1-\hat y)] Binary classifier loss
KL divergence KL(P∥Q)=∑xP(x)log⁡P(x)Q(x)\mathrm{KL}(P\|Q) = \sum_x P(x) \log\dfrac{P(x)}{Q(x)} VAE regularisation, RLHF penalty, drift detection
Identity H(P,Q)=H(P)+KL(P∥Q)H(P, Q) = H(P) + \mathrm{KL}(P\|Q) Explains why minimising cross-entropy minimises KL
Mutual information I(X;Y)=H(X)−H(X∣Y)=H(Y)−H(Y∣X)I(X;Y) = H(X) - H(X\mid Y) = H(Y) - H(Y\mid X) Feature selection, representation learning

API / Function Summary

scipy.stats.entropy(pk, qk=None, base=None, axis=0): returns entropy of pk, or KL(pk∥qk)\mathrm{KL}(pk \| qk) if qk is given. pk need not be pre-normalised. base=None gives nats; base=2 gives bits.

h = scipy.stats.entropy([0.5, 0.5], base=2)   # 1.0 bit
kl = scipy.stats.entropy(p, q, base=2)         # KL(p || q) in bits

log_loss, called as sklearn.metrics.log_loss(y_true, y_pred, labels=None, normalize=True), returns the cross-entropy (log) loss. y_pred is predicted probabilities, shape (n_samples,) or (n_samples, n_classes).

loss = log_loss(y_true, y_pred_proba)

sklearn.feature_selection.mutual_info_classif(X, y, discrete_features='auto', n_neighbors=3, random_state=None) (and mutual_info_regression for continuous targets): returns an ndarray of shape (n_features,), estimated mutual information with the target in nats, non-negative.

mi = mutual_info_classif(X, y, random_state=42)
ranking = (
    pd.Series(mi, index=X.columns).sort_values(ascending=False)
)

Algorithm / Approach Comparison

Approach Best for Weakness Function
Correlation (Pearson) Quick linear-relationship screening Misses non-linear and non-monotonic relationships entirely df.corr(), np.corrcoef
Mutual information Any relationship, linear or not Needs more data to estimate reliably; no fixed scale for “how strong” mutual_info_classif / _regression
Cross-entropy / log loss Classifier training loss and evaluation Only defined for probabilistic predictions, sensitive to miscalibration log_loss, framework loss functions
KL divergence Comparing two distributions directly (drift, VAE, RLHF) Asymmetric, undefined if support doesn’t match scipy.stats.entropy(p, q)

Decision Guide

Do you need to rank features by predictive power?
|-- Yes
|   |-- Only linear relationships, want speed?
|   |   → correlation
|   `-- Non-linear relationships, or unsure?
|       → mutual_info_classif/regression
`-- No
    |-- Training/evaluating a classifier?
    |   → cross-entropy / log_loss
    `-- Comparing two known distributions
        (drift, prior vs. posterior)? → KL divergence

Metrics at a Glance

Metric Range Higher/lower is “better”? Use when
Entropy H(P)H(P) 00 to log⁡(K)\log(K) Context-dependent (not a “quality” metric) Measuring how uncertain a distribution is
Cross-entropy H(P,Q)H(P,Q) H(P)H(P) to ∞\infty Lower is better Classifier training loss
KL divergence 00 to ∞\infty Lower (closer to 0) means more similar Distribution comparison, drift detection
Mutual information 00 to the smaller of H(X)H(X) and H(Y)H(Y) (discrete variables) Higher means more informative Feature selection ranking

Numerical Pitfalls Cheat Sheet

Pitfall Symptom Fix
log⁡(0)\log(0) nan or -inf loss Clip probabilities to [ϵ,1−ϵ][\epsilon, 1-\epsilon], e.g. ϵ=10−12\epsilon = 10^{-12}
Unnormalised distribution Entropy/KL values don’t match expectations Divide by .sum() before computing, or let scipy.stats.entropy normalise for you
qq has a zero where pp doesn’t KL(P∥Q)=∞\mathrm{KL}(P\|Q) = \infty Smooth qq with a small epsilon, or use a KL variant with support handling
Comparing MI scores across different datasets Misleading “this feature is better” conclusions Only rank features within the same run/dataset

AWS SageMaker Quick Reference

Task SageMaker approach Documentation
Mutual-information feature selection at scale SKLearnProcessor running a script that calls mutual_info_classif SageMaker Processing SDK
Monitoring prediction distribution drift SageMaker Model Monitor (built-in data quality / drift checks) SageMaker Model Monitor docs
Cross-entropy training loss on GPU Any Estimator (PyTorch/TensorFlow): loss defined in your training script SageMaker Training Jobs

Chapter 6 — Python Data Science Ecosystem

Key Terms

Term Definition
Broadcasting NumPy applying an operation across arrays of compatible shapes without an explicit loop
Series A 1D labelled array: a single pandas column
DataFrame A 2D labelled table, potentially with different types per column
Index The row-label structure that lets pandas align data across sources
Estimator A scikit-learn object (model or transformer) that follows the .fit()/.predict()/.transform() API

Library Purpose Table

Library Role When you need it
numpy Fast N-dimensional arrays, maths Any numerical computation, linear algebra
pandas Labelled tabular data Loading, cleaning, transforming, grouping datasets
matplotlib Low-level plotting Custom charts, fine-grained control
seaborn Statistical visualisation EDA, distributions, categorical comparisons
scipy Scientific computing Statistical tests, optimisation, signal processing
scikit-learn ML algorithms + tooling Models, preprocessing, cross-validation, metrics
jupyter Interactive notebooks Exploration, prototyping, sharing results

NumPy Quick Reference

np.array([1, 2, 3]); np.zeros((3, 4)); np.ones((2, 3)); np.eye(4)
np.arange(start, stop, step); np.linspace(start, stop, n)
np.random.randn(n); np.random.randint(low, high, size)

a.shape; a.dtype; a.ndim; a.size

a + b, a - b, a * b, a / b        # element-wise (broadcasting)
a @ b                              # matrix multiply
a.T                                # transpose
np.sum(a, axis=0)                  # sum along axis
np.mean(a), np.std(a), np.var(a)
a[a > threshold]                   # boolean indexing
np.where(condition, x, y)          # conditional selection
a.reshape(3, 4); a.flatten(); a.ravel()

pandas Quick Reference

pd.read_csv('file.csv'); pd.read_parquet('file.parquet')
df.shape; df.dtypes; df.head(n); df.info(); df.describe()
df.isnull().sum(); df.nunique()

df['col']; df[['col_a', 'col_b']]
df.loc[label]; df.iloc[n]
df[df['col'] > value]; df.query("col > @value")

df['new'] = df['a'] + df['b']
df.rename(columns={'old': 'new'}); df.drop(columns=['col'])
df.sort_values('col', ascending=False)
df.drop_duplicates(subset=['col'])
df.fillna(value); df.dropna(subset=['col'])

df.groupby('col')['val'].mean()
df.groupby(['a', 'b'])['val'].agg(['mean', 'count', 'std'])
df.pivot_table(values='v', index='r', columns='c', aggfunc='mean')

matplotlib / seaborn Quick Reference

fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(x, y, color='#4E79A7', linewidth=2, label='name')
ax.scatter(x, y, s=50, alpha=0.5, c=colours, cmap='viridis')
ax.bar(categories, values, color='steelblue', edgecolor='white')
ax.set_title('Title'); ax.set_xlabel('X'); ax.set_ylabel('Y')
ax.legend()
plt.savefig('out.png', dpi=150, bbox_inches='tight')

sns.histplot(data=df, x='col', hue='group', kde=True, bins=30)
sns.boxplot(data=df, x='cat', y='val', hue='group')
sns.barplot(data=df, x='cat', y='val', errorbar='sd')
sns.scatterplot(data=df, x='a', y='b', hue='label', alpha=0.5)
sns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm', center=0)
sns.pairplot(df, hue='target')

scikit-learn API Pattern

model.fit(X_train, y_train)
y_pred = model.predict(X_test)
score  = model.score(X_test, y_test)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)  # fit on train
X_test_s = scaler.transform(X_test)  # apply to test (no fit!)

from sklearn.pipeline import Pipeline
pipe = Pipeline([
    ('scale', StandardScaler()),
    ('model', LogisticRegression()),
])
pipe.fit(X_train, y_train)

Common Imports Block

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

from sklearn.model_selection import (
    train_test_split,
    cross_val_score,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.metrics import accuracy_score, classification_report
from sklearn.pipeline import Pipeline

plt.rcParams['figure.figsize'] = (10, 5)
sns.set_style('whitegrid')
np.random.seed(42)

Troubleshooting Quick Reference

Error / symptom Most likely cause Fix
ModuleNotFoundError: No module named 'numpy' Wrong environment active Activate the right env, then reinstall
ValueError: Input contains NaN sklearn received missing values Clean data first (Chapter 7)
DataConversionWarning: A column-vector y was passed y is shape (n,1) not (n,) y.ravel() or y.squeeze()
Plot is blank / not showing plt.show() before data calls Move data calls above plt.show()
seaborn hue not working Column has NaN values df.dropna(subset=['hue_col']) first

Chapter 7 — Data Cleaning & Wrangling

Key Terms

Term Definition
MCAR Missing Completely At Random: missingness unrelated to any variable, observed or not
MAR Missing At Random: missingness depends on observed variables, not the missing value
MNAR Missing Not At Random: missingness depends on the missing value. Naive imputation biases results
Winsorising Capping values beyond a percentile threshold at that threshold, rather than removing them
IQR Interquartile range, Q3−Q1Q_3 - Q_1; used to define robust, skew-resistant outlier bounds

Data Quality Problem Types

Problem Detection Treatment
Missing values df.isnull().sum() Impute, drop, or flag
Outliers IQR, z-score, domain knowledge Cap, remove, transform, or flag
Duplicates df.duplicated().sum() drop_duplicates()
Wrong types df.dtypes pd.to_numeric(), pd.to_datetime(), .astype()
Inconsistent categories .value_counts() .str.strip().str.lower() + .map()
Encoding errors Scan .value_counts() for a repeated Ã-plus-character pattern On rows containing à only: .str.encode('latin1').str.decode('utf-8') (reverses a UTF-8-read-as-Latin-1 mismatch)
Schema violations Rule-based checks (Procedure 7.6), not just .dtypes Drop, correct, or impute, once the rule that was broken is known

Missing Value Imputation Comparison

Strategy When to use Risk
Drop rows < 5% missing, MCAR Loses data
Mean fill Numeric, symmetric Sensitive to outliers
Median fill Numeric, skewed Slightly less precise than mean for normal data
Mode fill Categorical Can over-represent majority class
Forward fill Time series Wrong if gap is long
KNN imputation Other features correlate with the missing one Slow on large datasets
Model-based MAR, with strong predictors of the missing column Added complexity; risk of leakage if fit on the full dataset

Outlier Detection Methods

Method Formula Best for
IQR Outside Q1−1.5⋅IQRQ_1 - 1.5 \cdot \mathrm{IQR} or Q3+1.5⋅IQRQ_3 + 1.5 \cdot \mathrm{IQR} Skewed distributions, robust
Z-score ∣z∣>3\lvert z \rvert > 3, z=(x−μ)/σz = (x-\mu)/\sigma Roughly normal distributions
Percentile Below p1p_1 or above p99p_{99} Any distribution, simple
Domain cap Hard limits from business rules When the valid range is known a priori
Isolation Forest Tree-based anomaly score Multivariate, complex patterns (Chapter 19)

dtype Conversion Table

From To Method
Text (string number) float64 pd.to_numeric(col, errors='coerce')
Text (string number) Int64 (nullable int) pd.to_numeric(col).astype('Int64')
Text (date string) datetime64 pd.to_datetime(col, errors='coerce')
float64 int64 (after removing NaN) .dropna().astype(int)
Text category .astype('category')
int (0/1) bool .astype(bool)
datetime64 Text (formatted string) .dt.strftime('%Y-%m-%d')

Cleaning Checklist

[ ] 1. Check shape (rows x cols)
[ ] 2. Check dtypes: fix wrong types immediately
[ ] 3. Count missing values per column
[ ] 4. Count and remove exact duplicates
[ ] 5. Inspect category columns: normalise strings
[ ] 6. Detect numeric outliers (IQR or z-score)
[ ] 7. Decide imputation strategy per column
[ ] 8. Apply imputation
[ ] 9. Validate: assert on domain rules
[ ] 10. Compare before/after distributions visually

Troubleshooting Common Errors

Error / symptom Most likely cause Fix
ValueError: Input contains NaN (sklearn) Missing values not imputed Check df.isnull().sum(), impute first
Category encoding produces 20+ categories Inconsistent string values Normalise with .str.lower().str.strip() + map
KNN imputer runs very slowly Too many columns included Include only numeric columns relevant to the target
Duplicates reappear after a merge Join key not unique on one side Check key uniqueness first, or pass validate='one_to_one' to merge()
pd.to_datetime returns all NaT Unexpected date format Specify format= explicitly, e.g. format='%d/%m/%Y'

This chapter has no AWS SageMaker Quick Reference. Chapter 7’s cleaning operations are pandas/scikit-learn work performed before any AWS-side step. The SageMaker Feature Store material in Appendix A’s Chapter 2 entry is where a cleaned dataset like this chapter’s next gets registered.

Chapter 8 — SQL for Data Science

Key Terms

Term Definition
Common table expression (CTE) A named subquery defined with WITH, used for readability, not performance
Window function A function computed across rows related to the current row, without collapsing them the way GROUP BY does
Partition The PARTITION BY grouping a window function restarts its calculation within, analogous to GROUP BY but non-collapsing
Fact table The table of events/transactions being enriched (e.g. orders)
Dimension table The table of descriptive attributes joined onto a fact table (e.g. customers, products)

Window Function Syntax

function_name() OVER (
    PARTITION BY col1, col2   -- restart calculation for each group
    ORDER BY col3             -- row order within the partition
    ROWS BETWEEN n PRECEDING  -- optional: window frame
              AND CURRENT ROW
)

Window Function Quick Reference

Function Returns Example
ROW_NUMBER() Unique sequential integer (no ties) ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC)
RANK() Rank with gaps on ties (1,1,3) RANK() OVER (ORDER BY score DESC)
DENSE_RANK() Rank without gaps (1,1,2) DENSE_RANK() OVER (ORDER BY score DESC)
NTILE(n) Bucket into nn groups (quartiles etc.) NTILE(4) OVER (ORDER BY salary)
LAG(col, n) Value of col nn rows back LAG(revenue, 1) OVER (ORDER BY date)
LEAD(col, n) Value of col nn rows ahead LEAD(revenue, 1) OVER (ORDER BY date)
FIRST_VALUE(col) First value in partition FIRST_VALUE(order_date) OVER (PARTITION BY customer_id ORDER BY order_date)
LAST_VALUE(col) Last value in partition Requires ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
SUM(col) OVER (...) Running total (add ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW if dates can tie) SUM(rev) OVER (ORDER BY date)
AVG(col) OVER (...) Moving / partition average AVG(rev) OVER (PARTITION BY dept)
COUNT(*) OVER () Grand total row count Same value on every row

Aggregation Patterns

-- Basic group aggregation
SELECT col, COUNT(*), SUM(val), AVG(val), MIN(val), MAX(val)
FROM table GROUP BY col ORDER BY SUM(val) DESC;

-- Filter groups with HAVING
GROUP BY col HAVING SUM(val) > 1000

-- Multiple grouping levels
GROUP BY ROLLUP(region, dept)   -- subtotals at each level

-- Conditional aggregation
SELECT
    SUM(CASE WHEN status = 'completed'
             THEN revenue ELSE 0 END) AS completed_rev,
    SUM(CASE WHEN status = 'returned'
             THEN revenue ELSE 0 END) AS returned_rev
FROM orders;

CTE Template

WITH
cte_name_1 AS ( SELECT ... ),
cte_name_2 AS ( SELECT ... FROM cte_name_1 WHERE ... ),
cte_name_3 AS (
    SELECT
        *,
        RANK() OVER (PARTITION BY ... ORDER BY ...) AS rank
    FROM cte_name_2
)
SELECT * FROM cte_name_3 WHERE rank <= 10;

Join Types

Join Rows kept
INNER JOIN Only rows matching in both tables
LEFT JOIN All rows from left, matched rows from right (NULLs where no match): the dominant join in analytics
RIGHT JOIN Mirror of LEFT JOIN (rare: reorder tables and use LEFT JOIN instead)
FULL OUTER JOIN All rows from both tables, NULLs where no match on either side
CROSS JOIN Every combination of left × right rows (use carefully)
-- Most common analytical join pattern
FROM fact_table f
LEFT JOIN dim_table d ON f.foreign_key = d.primary_key

Date Functions (DuckDB / ANSI SQL)

DATE_TRUNC('month', date_col)          -- first day of month
DATE_TRUNC('year', date_col)           -- first day of year
DATE_PART('month', date_col)           -- extract month number
DATEDIFF('day', start_date, end_date)  -- days between dates
date_col + INTERVAL '30 days'          -- add 30 days
date_col + INTERVAL '1 month'          -- add 1 month
CURRENT_DATE                           -- today

DuckDB vs sqlite3 vs pandas.query

Feature DuckDB sqlite3 pandas.query
Window functions Full support Yes (SQLite 3.25+) No
CTEs Yes Yes No
Multi-table JOINs Yes Yes No
Reads pandas DataFrames Direct Via manual insert N/A: IS pandas
Parallelism Multi-core Single-threaded Partial (numpy)
Aggregation on millions of rows Fast Slow Slow
Setup pip install duckdb stdlib Part of pandas

Common Analytical Query Skeletons

-- Top-N per group
WITH ranked AS (
    SELECT
        *,
        RANK() OVER (
            PARTITION BY group_col ORDER BY metric DESC
        ) AS r
    FROM table
)
SELECT * FROM ranked WHERE r <= 3;

-- Period-over-period comparison
WITH monthly AS (
    SELECT
        DATE_TRUNC('month', date_col) AS m,
        SUM(val) AS rev
    FROM t
    GROUP BY 1
)
SELECT
    m,
    rev,
    LAG(rev) OVER (ORDER BY m) AS prev,
    (rev / LAG(rev) OVER (ORDER BY m) - 1) * 100 AS pct
FROM monthly;

-- Running total
SELECT
    date_col,
    val,
    SUM(val) OVER (
        ORDER BY date_col
        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS running_total
FROM t;

-- Deduplicate (keep latest per ID)
WITH ranked AS (
    SELECT
        *,
        ROW_NUMBER() OVER (
            PARTITION BY id ORDER BY updated_at DESC
        ) AS r
    FROM t
)
SELECT * FROM ranked WHERE r = 1;

SQL vs pandas Decision Guide

Use SQL when Use pandas when
Data already lives in a database or warehouse Data is already loaded as a DataFrame
Query joins more than two tables The transformation is custom, specific to feature engineering
Aggregation runs over rows too numerous to hold in memory Task is quick exploration of a CSV never destined for a database

AWS Athena / Redshift Quick Reference

Task AWS approach Notes
Query Parquet/CSV directly from S3 Amazon Athena (serverless SQL) CREATE EXTERNAL TABLE ... LOCATION 's3://...'
Data warehouse queries at scale Amazon Redshift Window function syntax very close to DuckDB’s
Prefer over local pandas + DuckDB Once data lives in S3 and is too large for memory Billed per byte of data scanned

Chapter 9 — Exploratory Data Analysis & Visualisation

Key Terms

Term Definition
Univariate analysis Examining one variable in isolation (distribution, spread, outliers)
Bivariate analysis Examining the relationship between two variables (e.g. feature vs. target)
Multivariate analysis Examining relationships among three or more variables at once
Skewness Asymmetry of a distribution; positive = long right tail, negative = long left tail
Kurtosis “Tailedness” of a distribution; high kurtosis = more extreme outliers than a normal distribution
Cardinality Number of unique values in a categorical column
Data snooping Drawing conclusions from EDA that implicitly overfit to the full dataset, including the eventual test set

Dataframe Profiling Cheat Sheet

df.shape                       # (rows, columns)
df.dtypes                      # column types
df.info()                      # types + non-null counts + memory
# Numeric summary: count, mean, std, min, 25/50/75%, max
df.describe()
# Categorical summary: count, unique, top, freq
df.describe(include='object')
df.isnull().sum()              # missing count per column
df.duplicated().sum()          # duplicate row count
df.nunique()                   # unique value count per column
# Random rows (better than .head() for spotting surprises)
df.sample(5)
df.select_dtypes(include='number')  # numeric columns only
df.select_dtypes(include='object')  # categorical columns only

Plot Type Decision Guide

What are you looking at?
|-- One numeric variable
|   |-- Shape/skew/modality?
|   |   -> histogram + KDE (sns.histplot(kde=True))
|   |-- Quartiles/outliers only? -> boxplot
|   `-- Shape AND quartiles? -> violin plot
|-- One categorical variable
|   `-- Frequency of each level?
|       -> bar chart (value_counts().plot(kind='bar'))
|-- Two numeric variables
|   |-- Relationship shape? -> scatter plot
|   `-- Strength across many pairs?
|       -> correlation heatmap (sns.heatmap(df.corr()))
|-- One numeric + one categorical
|   `-- Distribution shifts by group?
|       -> boxplot/violin split by category, or KDE
|-- Two categorical variables
|   `-- Association?
|       -> cross-tab + stacked bar, or pivot heatmap
`-- 3+ numeric variables
    `-- Full pairwise structure?
        -> pairplot (sns.pairplot, hue=target)

Plot Type Comparison

Plot Shows Best for Weakness
Histogram Frequency by value bucket Distribution shape, skew, modality Bin width choice changes the story
KDE plot Smoothed density estimate Comparing shapes of 2+ groups Can smooth away real bimodality if bandwidth too wide
Boxplot Median, quartiles, outliers Quick outlier/spread comparison across many groups Hides bimodality entirely
Violin plot Boxplot + KDE combined Comparing full shape across groups Misleading with very small groups (n < ~20)
Scatter plot Raw relationship between 2 numeric vars Spotting non-linear relationships, clusters Overplotting with large N (use alpha= or sample)
Correlation heatmap Pearson/Spearman r for all pairs Scanning many features for redundancy at once Linear (Pearson) misses non-linear relationships
Pairplot Scatter + histogram grid for all pairs Full-dataset overview in one figure Slow / unreadable beyond ~6 columns
Bar chart Count or aggregate by category Categorical frequency, group comparisons Misleading if categories aren’t sorted or truncated

Key Summary Statistics

Statistic Formula / call What it tells you
Mean series.mean() Central tendency, sensitive to outliers
Median series.median() Central tendency, robust to outliers
Mode series.mode() Most frequent value (numeric or categorical)
Std. deviation series.std() Typical spread around the mean
IQR Q3−Q1Q_3 - Q_1 Robust spread (middle 50% of data)
Skewness series.skew() >0>0 right-skewed, <0<0 left-skewed, ≈0\approx 0 symmetric
Kurtosis series.kurt() >0>0 heavier tails than normal, <0<0 lighter
Pearson rr series_a.corr(series_b) Linear relationship strength, −1-1 to 11
Spearman rr series_a.corr(series_b, method='spearman') Monotonic (linear or not) relationship strength

The IQR-rule outlier bounds are:

[ Q1−k⋅IQR, Q3+k⋅IQR ],k=1.5 (typical)[\,Q_1 - k \cdot \text{IQR},\ Q_3 + k \cdot \text{IQR}\,], \quad k = 1.5 \text{ (typical)}

The z-score outlier rule flags any value where:

∣x−xˉσ∣>3\left|\frac{x - \bar{x}}{\sigma}\right| > 3

Here σ\sigma is the population standard deviation (ddof=0), which is what scipy.stats.zscore uses. With nn values, no ∣z∣|z| can exceed n−1\sqrt{n-1}, so the rule can never fire when n≤10n \le 10.

Outlier Detection Methods

Method Assumes sklearn / scipy
IQR rule Nothing about distribution shape Manual (pandas)
Z-score Roughly normal distribution scipy.stats.zscore
Isolation Forest Multivariate anomalies sklearn.ensemble.IsolationForest
Local Outlier Factor Local density variation matters sklearn.neighbors.LocalOutlierFactor

Common Imports

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

pd.set_option('display.max_columns', 50)
sns.set_style('whitegrid')
plt.rcParams['figure.figsize'] = (10, 5)

AWS SageMaker Quick Reference

Task SageMaker approach Notes
Automated data quality report SageMaker Data Wrangler -> “Data Quality and Insights Report” Flags missing values, imbalance, target leakage, outliers automatically
Interactive EDA at scale SageMaker Studio notebook + pandas/seaborn on data from S3/Athena Same code as local, larger instance for bigger data
Distributed profiling for very large datasets SageMaker Processing job running a Spark/Deequ profiling script Needed once data exceeds single-notebook memory
Feature-target correlation / leakage detection Data Wrangler “Quick Model” analysis Trains a throwaway model to estimate feature importance during EDA

Chapter 10 — The Data Science Lifecycle

Key Terms

Term Definition
CRISP-DM Cross-Industry Standard Process for Data Mining: business understanding, data understanding, data preparation, modelling, evaluation, deployment
TDSP Microsoft’s Team Data Science Process: adds explicit deployment, monitoring, and governance emphasis CRISP-DM treats lightly
Project charter A lightweight, whole-project companion to the Chapter 2 problem statement, naming stage owners and kill criteria
Pre-mortem A structured exercise assuming the project already failed, to surface risks while still cheap to fix
Kill criteria The explicit condition under which a project is cancelled rather than continued on sunk cost
Loop-back A lifecycle transition to an earlier stage, triggered by what a later stage reveals: the norm, not the exception

The Eight Stages, Mapped to This Book

Stage What happens Where it lives
1. Problem framing Translate a business ask into a precise ML problem statement Chapter 2
2. Data collection & understanding Identify sources, pull data, understand structure/quality/limitations Chapters 7-9
3. Data preparation Clean, transform, engineer features into model-ready form Chapters 7, 14
4. Modelling Select algorithm family, train candidates, tune hyperparameters Chapters 11-20
5. Evaluation Measure against the stage-1 success metric: business metric, not just accuracy Chapters 13, 16
6. Deployment Package the model and serve predictions: batch or real-time Book 2, Chapter 23
7. Monitoring Watch for data drift, model drift, operational failures Book 2, Chapter 25
8. Iteration Feed evaluation/production learnings back into a revised problem statement Back to stage 1

Framework Comparison

Stage (this curriculum) CRISP-DM Microsoft TDSP
1. Problem framing Business Understanding Business Understanding
2. Data collection & understanding Data Understanding Data Acquisition & Understanding
3. Data preparation Data Preparation Data Acquisition & Understanding (cleaning); Modeling (feature engineering)
4. Modelling Modeling Modeling
5. Evaluation Evaluation Modeling (evaluation sub-phase)
6. Deployment Deployment Deployment
7. Monitoring (not explicit) Deployment (ongoing) / Customer Acceptance
8. Iteration (implicit feedback loop) Explicit: loops back to Business Understanding

Key difference: CRISP-DM (1996, data-mining era) treats deployment lightly and has no explicit monitoring phase. TDSP (2016, cloud/MLOps era) treats deployment, monitoring, and governance as first-class stages. This book follows TDSP’s operational emphasis.

Stage Checklist: Inputs, Outputs, Exit Criteria

Stage Inputs Outputs Exit criteria
1. Problem framing Stakeholder ask Filled problem statement Label, features, horizon, success metric signed off
2. Data understanding Raw data sources EDA report Data quality, coverage, label reliability documented
3. Data preparation Raw/understood data Model-ready dataset No known leakage; split defined; pipeline reproducible
4. Modelling Model-ready dataset Trained candidate model(s) At least one model beats the baseline
5. Evaluation Trained model(s), held-out test set Evaluation report Model metric AND business metric both meet stage-1 thresholds
6. Deployment Approved model Live serving endpoint or batch job Model reachable in production; rollback plan exists
7. Monitoring Live system Drift/performance dashboards + alerts Named owner receives alerts; retraining trigger defined
8. Iteration Monitoring signals, retrospective Revised problem statement Feeds back into stage 1

Common Failure Modes by Stage

Stage Failure mode Early warning sign
Problem framing No kill criteria; ML used where a rule would do “We’ll figure out success metrics later”
Data understanding Label doesn’t reflect current business process Label definition changed within the historical window
Data preparation Leakage; feature not available at prediction time Suspiciously high correlation with label (>0.9>0.9)
Modelling Optimising the wrong metric Model metric improves but no business change follows
Evaluation One-off evaluation, never repeated No scheduled re-evaluation cadence
Deployment Never leaves the notebook No named deployment owner in the charter
Monitoring No one owns production health Alerts routed to no one, or an unread inbox
Iteration “Done” treated as permanent No retrospective run; no revised problem statement

Decision Guide: Should This Be a Full Lifecycle Project?

Does a simple rule already meet the required performance?
|-- Yes → Ship the rule. Not a lifecycle project.
`-- No
    |-- One-off analysis, no deployed system?
    |   |-- Yes → EDA/analytics task, not
    |   |        a full lifecycle project.
    |   `-- No
    |       |-- Does historical labelled data exist
    |       |   (or can it be constructed cheaply)?
    |       |   |-- No → Data collection project first.
    |       |   |        Revisit once labels exist.
    |       |   `-- Yes → Full lifecycle project.
    |       |            Use the charter template.

Lifecycle Stage Owners (RACI Shorthand)

Stage Typically Responsible Typically Accountable
Problem framing Data scientist / SA Business sponsor
Data understanding Data scientist / data engineer Data scientist
Data preparation Data scientist / ML engineer Data scientist
Modelling Data scientist Data scientist
Evaluation Data scientist Business sponsor (signs off on business metric)
Deployment ML engineer / MLOps Engineering lead
Monitoring ML engineer / on-call Named individual, not “the team”
Iteration Whole team Business sponsor

AWS SageMaker Quick Reference

Lifecycle stage AWS service Notes
Problem framing (no AWS service: human process) Document in the charter
Data collection & understanding S3, Glue, Athena, SageMaker Data Wrangler Chapters 7-9 tooling maps here
Data preparation SageMaker Data Wrangler, SageMaker Feature Store Chapter 14
Modelling SageMaker Training Jobs, SageMaker Experiments Chapters 11-20
Evaluation SageMaker Clarify, SageMaker Model Monitor (baseline) Chapter 16; Book 2, Chapter 21 (Clarify)
Deployment SageMaker Endpoints (real-time), SageMaker Batch Transform Book 2, Chapter 23
Monitoring SageMaker Model Monitor, CloudWatch Book 2, Chapter 25
Iteration / orchestration SageMaker Pipelines, SageMaker Projects Book 2, Chapter 24

Chapter 11 — Supervised Learning: Overview of Approaches

Key Terms

Term Definition
Inductive bias An algorithm family’s built-in assumption about the shape of the relationship between inputs and outputs
Bias (statistical) How wrong a model’s assumptions are, systematically, no matter how much data it gets
Variance (statistical) How much a model’s predictions change when trained on a different sample from the same population
Curse of dimensionality As dimensionality grows, all points become roughly equidistant, so “nearest neighbour” stops being meaningful
No Free Lunch theorem Averaged over all possible problems, no algorithm outperforms any other. It does not rule out algorithms that reliably win on the specific kinds of problems seen in practice
Conditional independence Naive Bayes’ assumption that features are independent of each other given the class label

Algorithm Comparison Table

Algorithm Task Interpretable Non-linear Needs scaling Missing values sklearn class
Logistic Regression Classification Yes No (linear) Yes No LogisticRegression
Linear Regression Regression Yes No No (yes once regularised) No LinearRegression
Ridge Regression Regression Yes No Yes No Ridge
Decision Tree Both Yes (if shallow) Yes No Yes (sklearn 1.3+) DecisionTreeClassifier / Regressor
K-Nearest Neighbours Both No Yes Yes (critical) No KNeighborsClassifier / Regressor
Naive Bayes Classification Partial No No No GaussianNB, MultinomialNB
Random Forest Both No Yes No Yes (sklearn 1.4+) RandomForestClassifier / Regressor
SVM Both No Yes (with kernel) Yes (critical) No SVC, SVR
XGBoost Both No Yes No Yes (native) xgboost.XGBClassifier

The Universal sklearn API

model.fit(X_train, y_train)   # train
# Class labels (classification) or values (regression)
model.predict(X_test)
# Class probabilities (classification only)
model.predict_proba(X_test)
model.score(X_test, y_test)   # accuracy (clf) or R² (reg)

# Pipeline pattern (always use this)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model', YourClassifier())
])
pipe.fit(X_train, y_train)
pipe.predict(X_test)

Bias-Variance Quick Reference

Symptom Diagnosis Fix
Train accuracy high, test accuracy low High variance (overfitting) More data, regularisation, simpler model, ensemble
Both train and test accuracy low High bias (underfitting) More complex model, better features, less regularisation
Large gap between CV folds High variance Reduce model complexity, more training data

Algorithm Selection Decision Tree

What is your target?
|-- Discrete category → Classification
|   |-- Need interpretability?
|   |   |-- Yes → LogisticRegression (linear) or
|   |   |         DecisionTree (max_depth=4)
|   |   `-- No  → RandomForest or XGBoost
|   |             (usually best on tabular)
|   `-- Text data? → MultinomialNB or fine-tuned
|       transformer
`-- Continuous number → Regression
    |-- Linear relationship? → LinearRegression / Ridge
    |-- Nonlinear? → RandomForestRegressor or XGBRegressor
    `-- Time series? → Specialised forecasting models

Key Hyperparameters to Know

Class Key hyperparams Notes
LogisticRegression C (inverse regularisation, default 1.0), max_iter Larger C = less regularisation
DecisionTreeClassifier max_depth, min_samples_leaf, min_samples_split Limit depth to prevent overfitting
KNeighborsClassifier n_neighbors (default 5), metric, algorithm More neighbours = smoother boundary
RandomForestClassifier n_estimators, max_depth, max_features Start with n_estimators=100
GaussianNB var_smoothing Rarely needs tuning

Common Imports

from sklearn.linear_model import (
    LogisticRegression,
    LinearRegression,
    Ridge,
)
from sklearn.tree import (
    DecisionTreeClassifier,
    DecisionTreeRegressor,
)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.ensemble import (
    RandomForestClassifier,
    RandomForestRegressor,
)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import (
    train_test_split,
    cross_val_score,
)
from sklearn.metrics import (accuracy_score, classification_report,
                              confusion_matrix, f1_score,
                              mean_squared_error, r2_score)

AWS SageMaker Quick Reference

Task SageMaker approach Notes
Linear/logistic regression at scale SageMaker Linear Learner built-in algorithm Distributed training for large tabular datasets
Tree ensembles at scale SageMaker built-in XGBoost algorithm Managed, spot-instance-friendly training jobs
k-NN at scale SageMaker k-NN built-in algorithm Approximate nearest-neighbour index for large datasets
Comparing algorithm families quickly SageMaker Autopilot Runs several candidate algorithms and reports a leaderboard, mirroring Procedure 11.2

Chapter 12 — Ensemble Methods

Ensemble Method Comparison

Method Strategy Reduces Best for Library
Bagging Bootstrap samples, average Variance High-variance base learners BaggingClassifier
Random Forest Bagging + random features Variance Tabular data, robust baseline RandomForestClassifier/Regressor
sklearn Gradient Boosting Sequential residual fitting Bias + Variance General tabular, slower GradientBoostingClassifier/Regressor
XGBoost GB + regularisation, pruning, sparsity Bias + Variance Most tabular tasks xgboost.XGBClassifier/Regressor
LightGBM Leaf-wise GB, GOSS, EFB Bias + Variance Large datasets, speed-critical lightgbm.LGBMClassifier/Regressor
Stacking Meta-learner on base predictions Both Squeezing last % accuracy StackingClassifier/Regressor

Key Hyperparameters

Random Forest: n_estimators (100, range 100-500), max_depth (None), max_features ('sqrt'), min_samples_leaf (1, range 1-20), n_jobs=-1.

XGBoost: n_estimators (100, range 100-1000), learning_rate/eta (0.3, range 0.01-0.3), max_depth (6, range 3-8), subsample (1.0, range 0.6-1.0), reg_alpha/reg_lambda (L1/L2).

LightGBM: n_estimators (100, range 100-2000), learning_rate (0.1, range 0.01-0.1), num_leaves (31, range 20-150, primary complexity control), min_child_samples (20, range 10-100).

Feature Importance Types

Type How computed Strengths Weaknesses
Impurity (Gini/variance) Sum of impurity reduction at splits Fast, no extra data Biased towards high-cardinality features
Permutation Drop in score when feature shuffled No high-cardinality bias Slower, needs held-out data; misleads on correlated features
SHAP Shapley values Per-prediction explanations Requires shap library

Common Imports

from sklearn.ensemble import (
    RandomForestRegressor,
    GradientBoostingRegressor,
    StackingRegressor,
)
import xgboost as xgb
import lightgbm as lgb
from sklearn.inspection import permutation_importance

AWS SageMaker Quick Reference

Task SageMaker approach
Train XGBoost without a custom container Built-in xgboost algorithm image
Distributed training on large tabular data Built-in algorithm + multi-instance training job

Chapter 13 — Cross-Validation & Model Selection

CV Strategy Selection Guide

Situation Strategy sklearn class
Standard classification Stratified k-fold (k=5) StratifiedKFold
Standard regression k-Fold (k=5) KFold
Time-ordered data Time-series split TimeSeriesSplit
Tiny dataset (<50 samples) Leave-One-Out LeaveOneOut
Hyperparam tuning + evaluation Nested CV GridSearchCV inside cross_val_score
Groups (same subject in train+test) Group k-fold GroupKFold

Common CV Pitfalls

Pitfall Consequence Fix
Preprocessing fitted on whole data, not per-fold Data leakage: overoptimistic scores Always use Pipeline
Reporting test score after using it for selection Overoptimistic: selection bias Nested CV or a held-out final test set
Shuffled CV on time-series data Leaks future into training TimeSeriesSplit
Not stratifying for imbalanced classes Unstable fold scores StratifiedKFold

Scoring Strings

Scoring string Metric Task
'accuracy' Fraction correct Classification
'f1_weighted' Weighted F1 Multiclass
'roc_auc' ROC-AUC Binary classification
'r2' R² Regression
'neg_root_mean_squared_error' −RMSE Regression

All regression scorers are negated, since sklearn maximises. Negate the result to get the real metric.

How Many Folds?

Dataset size Recommendation
< 50 samples LeaveOneOut
50-500 samples k=10
500-10,000 samples k=5 (default)
> 10,000 samples k=3 or single split

Pipeline + CV Pattern (Always Use This)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import (
    cross_val_score,
    StratifiedKFold,
)

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model', YourModel()),
])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
    pipe, X, y, cv=cv, scoring='accuracy', n_jobs=-1
)

Common Imports

from sklearn.model_selection import (
    train_test_split,
    KFold,
    StratifiedKFold,
    TimeSeriesSplit,
    LeaveOneOut,
    cross_val_score,
    cross_val_predict,
    cross_validate,
    GridSearchCV,
    RandomizedSearchCV,
)

Chapter 14 — Feature Engineering

Key Terms

Term Definition
Feature A measurable property of an observation used as model input
Encoding Converting a categorical variable into numbers
Cardinality Number of unique values in a categorical column
Leakage A feature (or its fitted statistics) using information the model shouldn’t have at prediction time
Cyclical encoding sin/cos transform making a wrapping variable (hour, day of week) numerically continuous
Interaction feature A feature built from combining two or more other features

Encoding Strategy Comparison

Encoding Cardinality fit Leakage risk Model type
One-hot Low (< 20) None Any, especially linear/distance-based
Ordinal Low, ordered None Any
Integer codes (OrdinalEncoder) Any None Tree-based only
Frequency High None Any
Target / mean (CV) High High: must use cross-fitting Any

Scaler Comparison

Scaler Formula Sensitive to outliers? Use when
StandardScaler (x−μ)/σ(x-\mu)/\sigma Yes Default, roughly normal data
MinMaxScaler (x−min⁡)(x-\min) / (max⁡−min⁡)(\max-\min) Very Bounded input needed
RobustScaler (x−median)/IQR(x-\text{median})/\text{IQR} No Data has significant outliers
None — — Tree-based models

Decision Guide

Is the column categorical?
|-- Yes
|   |-- Cardinality < 20?
|   |   |-- Has natural order? → OrdinalEncoder
|   |   `-- No order           → OneHotEncoder
|   `-- Cardinality >= 20?
|       |-- Tree-based model? → OrdinalEncoder
|       `-- Linear/other model?
|           → Frequency or CV target encoding
`-- No (numeric)
    |-- Datetime?
    |   → extract hour/dow/month + cyclical sin/cos
    |-- Right-skewed? → log1p transform
    `-- Feeding linear/distance model?
        → StandardScaler / RobustScaler

Common Imports

from sklearn.preprocessing import (
    OneHotEncoder,
    OrdinalEncoder,
    StandardScaler,
    PolynomialFeatures,
    KBinsDiscretizer,
)
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

AWS SageMaker Quick Reference

Task SageMaker approach
No-code feature transforms SageMaker Data Wrangler flow
Shared feature store (train + serve) SageMaker Feature Store
Embed preprocessing in inference sklearn.Pipeline saved via joblib, deployed as one model

Chapter 15 — Feature Selection & Dimensionality Reduction

Key Terms

Term Definition
Feature selection Choosing a subset of existing features; the rest are discarded, survivors keep their original meaning
Feature extraction Constructing a smaller number of new features as combinations of the originals (e.g. PCA components)
Curse of dimensionality The set of problems (sparse data, meaningless distances, overfitting) that worsen as feature count grows
Explained variance ratio Fraction of total dataset variance captured by a given principal component
Loading The weight of an original feature in a principal component’s linear combination
Perplexity (t-SNE) Roughly, the effective number of nearest neighbours considered per point; controls local vs. global balance
Embedding The low-dimensional representation produced by PCA/t-SNE/UMAP

Selection Method Comparison

Method Family Model-aware? Handles non-linear relationships? Relative cost
Variance threshold Filter No N/A (variance only) Very low
Correlation with target Filter No No (linear only) Very low
Mutual information Filter No Yes Low-medium
RFE / RFECV Wrapper Yes Depends on base estimator High (retrains repeatedly)
L1 / Lasso coefficients Embedded Yes No (linear model) Low (one fit)
Tree feature importance Embedded Yes Yes Low-medium (one fit)

PCA vs. t-SNE vs. UMAP

Property PCA t-SNE UMAP
Linear or non-linear Linear Non-linear Non-linear
Preserves Global variance Local neighbourhood structure Local + more global structure than t-SNE
Deterministic Yes No (stochastic) No (stochastic, but more stable)
.transform() on new data Yes No (classic implementation) Yes (with caveats)
Interpretable components Somewhat (loadings) No No
Typical use Preprocessing for modelling, or visualisation Visualisation only Visualisation, sometimes clustering preprocessing
Relative speed on large data Fast Slow Faster than t-SNE

Decision Guide

Need the output to feed a downstream model (not a plot)?
|-- Yes
|   |-- Want to keep original feature meaning?
|   |   → Feature selection (filter/wrapper/embedded)
|   `-- OK losing interpretability for compactness?
|       → PCA
`-- No — just exploring / visualising
    |-- Fast look, large dataset (100k+ rows)? → UMAP
    |-- Best local cluster separation, small-medium
    |   dataset? → t-SNE
    `-- Fast, deterministic, reproducible 2D view? → PCA

API / Function Summary

# Features with variance <= threshold are removed
VarianceThreshold(threshold=0.0)

SelectKBest(
    # or f_classif, chi2, mutual_info_regression, f_regression
    score_func=mutual_info_classif,
    k=10,
)

mutual_info_classif(
    X, y, discrete_features='auto', random_state=42
)

RFE(estimator, n_features_to_select=10, step=1)
# .support_, .ranking_, .n_features_
RFECV(estimator, step=1, cv=5, scoring='accuracy')

SelectFromModel(estimator, prefit=False, threshold='median')

PCA(
    # int, float in (0,1) (variance target), or None (all)
    n_components=None,
    random_state=42,
    svd_solver='auto',
)
# .explained_variance_ratio_, .components_ (loadings),
# .n_components_

# Like PCA but for sparse matrices, no centering
TruncatedSVD(n_components=10, random_state=42)

# PCA for data with non-linear structure; the kernel trick
# from Chapter 11's SVM section, applied to PCA
KernelPCA(n_components=2, kernel='rbf', gamma=None, random_state=42)
# No simple inverse back to the original feature space

TSNE(
    n_components=2,
    perplexity=30,
    learning_rate='auto',
    random_state=42,
    max_iter=1000,
)
# No .transform() for new points in the classic implementation

UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
# supports .transform() on new data, unlike classic t-SNE

Common Imports

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import (
    VarianceThreshold,
    SelectKBest,
    mutual_info_classif,
    mutual_info_regression,
    RFE,
    RFECV,
    SelectFromModel,
)
from sklearn.decomposition import PCA, TruncatedSVD, KernelPCA
from sklearn.manifold import TSNE
from sklearn.ensemble import RandomForestClassifier
# pip install umap-learn
import umap

AWS SageMaker Quick Reference

Task SageMaker approach Documentation
Distributed PCA on very large datasets Built-in PCA algorithm sagemaker.PCA estimator
No-code feature importance / correlation analysis SageMaker Data Wrangler analyses sagemaker.wrangler.processing.DataWranglerProcessor
Interactive exploratory dimensionality reduction SageMaker Studio notebook with sklearn/UMAP Standard notebook instance / Studio kernel
Feature selection as a repeatable pipeline step SageMaker Processing job (SKLearnProcessor) wrapping the fitted selector sagemaker.sklearn.processing.SKLearnProcessor

Chapter 16 — Model Evaluation

Classification Metrics — Formulas

Metric Formula Higher = better?
Accuracy (TP+TN)/(TP+FP+FN+TN)(TP+TN)/(TP+FP+FN+TN) Yes
Precision TP/(TP+FP)TP/(TP+FP) Yes
Recall (Sensitivity) TP/(TP+FN)TP/(TP+FN) Yes
F1 Score 2PR/(P+R)2PR/(P+R) Yes
ROC-AUC Area under ROC curve Yes (0.5 = random)
PR-AUC / Avg Precision Area under PR curve Yes
Log Loss −mean(ylog⁡y^+(1−y)log⁡(1−y^))-\text{mean}(y\log\hat y + (1-y)\log(1-\hat y)) No (lower better)

Regression Metrics — Formulas

Metric Formula Sensitive to outliers?
MSE mean((y−y^)2)\text{mean}((y-\hat y)^2) Yes
RMSE MSE\sqrt{\text{MSE}} Yes
MAE mean(∣y−y^∣)\text{mean}(\lvert y-\hat y\rvert) Less
MAPE mean(∣y−y^∣/y)×100\text{mean}(\lvert y-\hat y\rvert/y)\times 100 Less
R2R^2 1−SSres/SStot1-SS_{res}/SS_{tot} Moderate

RMSE/MAE ratio > 2 indicates outliers are significantly influencing RMSE.

Metric Selection Guide

Business context Recommended metric
Balanced classes, symmetric errors Accuracy or F1
Imbalanced classes F1, PR-AUC
FP expensive (spam filter) Precision
FN expensive (cancer, fraud) Recall
Compare models before setting threshold ROC-AUC
Probabilities matter (pricing, risk) Log Loss + calibration
Rare event detection PR-AUC (ROC-AUC is misleading)

Calibration Interpretation

Curve shape Meaning Fix
On the diagonal Well-calibrated Nothing needed
S-curve (probabilities avoid 0 and 1) Under-confident at the extremes; common in Random Forests Isotonic regression or Platt (sigmoid) scaling
Below diagonal Over-estimates: observed rate is lower than predicted Isotonic regression or Platt (sigmoid) scaling

sklearn API Quick Reference

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score,
    average_precision_score,
    log_loss,
    confusion_matrix,
    classification_report,
    roc_curve,
    precision_recall_curve,
    mean_squared_error,
    mean_absolute_error,
    r2_score,
)
from sklearn.calibration import (
    calibration_curve,
    CalibratedClassifierCV,
)

AWS SageMaker Quick Reference

Task Approach
Log metrics during training sagemaker.Session().log_metric(name, value, step)
Automatic model quality monitoring SageMaker Model Monitor: ModelQualityMonitor

Chapter 17 — Hyperparameter Optimisation

Search Strategy Comparison

Strategy Pros Cons Use when
Manual (one-at-a-time) Simple, interpretable Misses interactions Debugging only
Grid Search Exhaustive over grid Exponential in #params ≤\leq 3 params, fast model
Random Search Wide coverage, parallelisable No guidance from past trials Most situations
Bayesian (Optuna) Sample-efficient, prunes bad trials Trials run in sequence Expensive models

Rule of thumb: random search with 50-100 trials beats grid search on most problems, especially once a grid is restricted to just a couple of values per dimension. This chapter’s Worked Example measures the gap directly and shows why: a coarse grid can only sample the corners of the search space, never the interior.

Key Hyperparameters by Algorithm

XGBoost/LightGBM: n_estimators (100-1000), max_depth (3-8), learning_rate (0.01-0.3, log scale), subsample (0.6-1.0), reg_alpha/reg_lambda.

Random Forest: n_estimators (100-1000), max_depth (5-30 or None), max_features ('sqrt', 'log2'), min_samples_leaf (1-10).

Logistic Regression: C (0.001-100, log scale), penalty ('l1', 'l2', 'elasticnet').

Optuna Quick-Start

import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int(
            'n_estimators', 50, 500
        ),
        'max_depth': trial.suggest_int('max_depth', 2, 10),
        'learning_rate': trial.suggest_float(
            'learning_rate', 0.01, 0.3, log=True
        ),
    }
    model = MODEL_CLASS(**params, random_state=42)
    scores = cross_val_score(
        model, X_train, y_train, cv=5, scoring='roc_auc'
    )
    return scores.mean()

study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42),
)
study.optimize(objective, n_trials=100)

Optuna suggest_* API

trial.suggest_int('name', low, high)
trial.suggest_float('name', low, high)
trial.suggest_float('name', low, high, log=True)
trial.suggest_categorical('name', ['a', 'b', 'c'])

Validation Set Hygiene

Training data → hyperparameter search (CV folds)
             → pick best config
Test data     → evaluate best model ONCE, at the very end

Never tune on the test set. After tuning, the CV score is optimistic. The test score is the honest estimate.

AWS SageMaker HPO Quick Reference

Feature SageMaker equivalent
Bayesian search strategy='Bayesian' in HyperparameterTuner
Random search strategy='Random'
Parallel trials max_parallel_jobs=N

Chapter 18 — Regularisation

Regularisation Method Comparison

Method Penalty Zeroes coefficients? Best for
Ridge (L2) α∑wi2\alpha \sum w_i^2 No: shrinks all Most features informative, correlated features
Lasso (L1) α∑∣wi∣\alpha \sum \lvert w_i\rvert Yes: sparse solution Many irrelevant features, automatic selection
Elastic Net α(ρ∑∣wi∣+12(1−ρ)∑wi2)\alpha(\rho\sum\lvert w_i\rvert+\tfrac{1}{2}(1-\rho)\sum w_i^2) Yes Correlated features + sparsity
Dropout Random zero-out during training N/A Neural networks
Early stopping Stop when val loss stops improving (with patience) N/A Neural nets, gradient boosting

Bias-Variance Trade-off

Low Complexity High Complexity
Bias High Low
Variance Low High
Test Error High (“underfitting”) High (“overfitting”)

Learning Curve Diagnosis

Pattern Diagnosis Fix
Both curves converge to a similarly high error High bias More flexible model, better features
Train error stays low, validation error stays higher, gap persists High variance More data, more regularisation

Key Parameters

Model Regularisation param Direction
Ridge / Lasso alpha Larger = more regularisation
ElasticNet alpha, l1_ratio l1_ratio: 0=Ridge, 1=Lasso
LogisticRegression C Smaller C = more regularisation (C=1/αC=1/\alpha)
XGBoost/LightGBM reg_alpha (L1), reg_lambda (L2) Larger = more

Regularisation in Other Model Types

Model type L2 equivalent L1 equivalent
XGBoost/LightGBM reg_lambda reg_alpha
Neural networks weight decay in optimiser None (dropout is the main regulariser)
Decision trees max_depth, min_samples_leaf (structural) —

sklearn API

from sklearn.linear_model import (
    Ridge, RidgeCV, Lasso, LassoCV, ElasticNet, ElasticNetCV,
    LogisticRegression, LogisticRegressionCV,
)

Dropout Quick Reference

import torch.nn as nn
# Zero out 50% of activations during training
nn.Dropout(p=0.5)
# model.train() -> dropout active
# model.eval() -> dropout disabled

AWS SageMaker Equivalent

from sagemaker.linear_learner import LinearLearner
ll = LinearLearner(
    predictor_type='binary_classifier', l1=0.01, wd=0.001
)

Chapter 19 — Unsupervised Learning

Key Terms

Term Definition
Clustering Grouping similar points without ground-truth labels
Anomaly detection Identifying points that don’t fit the pattern of the rest of the data
Centroid Mean position of all points assigned to a cluster (K-Means)
Inertia Sum of squared distances from each point to its assigned centroid
Core point A point with at least min_samples neighbours within eps (DBSCAN)
Contamination Expected fraction of anomalies in a dataset
BIC / AIC Bayesian/Akaike Information Criterion: single scores for choosing GMM component count; BIC penalises extra components more as data grows, AIC keeps the penalty fixed

Algorithm Comparison

Approach Best for Weakness
K-Means Fast, large datasets, roughly spherical clusters Must specify K; assumes equal-size, spherical clusters
Gaussian Mixture Model Elliptical clusters, soft/probabilistic membership Slower (full covariance per component); can still converge to a poor local optimum
DBSCAN Arbitrary-shaped clusters, automatic outlier detection Sensitive to eps/min_samples; struggles with varying density
Agglomerative Small datasets, exploratory analysis, dendrograms At least O(n2)O(n^2) time and memory: impractical above ~10k rows
Isolation Forest High-dimensional anomaly detection, scales well Random results across runs; needs contamination estimate

Clustering Evaluation Metrics

Metric Range Higher better? Use when
Silhouette score -1 to 1 Yes General-purpose cluster quality
Davies-Bouldin index 0 to ∞ No Comparing compactness/separation across K
Calinski-Harabasz index 0 to ∞ Yes Fast, good for large datasets
Adjusted Rand Index -0.5 to 1 Yes Only with ground-truth labels available

Decision Guide

Roughly know cluster count, likely spherical? → K-Means
Need soft membership or elliptical clusters? → Gaussian Mixture Model
Unknown K, or irregular shapes / noise? → DBSCAN
Small dataset (<10k), want to explore visually?
    → Agglomerative + dendrogram

High-dim, large anomaly-detection dataset?
    → IsolationForest
Density varies a lot across feature space?
    → LocalOutlierFactor
Small, clean dataset, tight non-linear boundary?
    → OneClassSVM

Common Imports

from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.mixture import GaussianMixture
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor, NearestNeighbors
from sklearn.metrics import (
    silhouette_score,
    davies_bouldin_score,
    adjusted_rand_score,
)

AWS SageMaker Quick Reference

Task SageMaker approach
Managed K-Means at scale sagemaker.KMeans built-in algorithm
Managed anomaly detection sagemaker.RandomCutForest built-in algorithm
Real-time anomaly scoring on streams Managed Service for Apache Flink calling a deployed model (Kinesis Data Analytics for SQL, with RANDOM_CUT_FOREST, retired in January 2026)

Chapter 20 — Time Series Analysis

Key Terms

Term Definition
Trend The long-run direction of the series
Seasonality A pattern that repeats at a fixed, known period
Stationarity Statistical properties (mean, variance, autocorrelation) constant over time
Differencing Subtracting a lagged value to remove trend or seasonality
ACF / PACF Autocorrelation / partial autocorrelation functions
Unit root Property of a non-stationary series; the ADF test’s null hypothesis

ACF / PACF → Model Order Cheat Sheet

Pattern Suggests
ACF cuts off sharply after lag q, PACF decays gradually MA(q)
PACF cuts off sharply after lag p, ACF decays gradually AR(p)
Both decay gradually Mixed ARMA(p,q): use auto_arima
ACF decays very slowly Non-stationary: difference first
Spike at lag m and multiples of m Seasonal component at period m

Forecasting Method Comparison

Approach Best for Weakness
Naive / seasonal naive Baseline: always compute first No real “modelling”
Exponential smoothing Simple, robust, minimal tuning Limited to trend + one seasonal pattern
ARIMA / SARIMA Single series, built-in prediction intervals Requires stationarity, manual/auto order selection
Lag-feature gradient boosting Many related series, external regressors No native uncertainty intervals

Forecast Evaluation Metrics

Metric Formula Use when
MAE 1n∑∣yi−y^i∣\frac{1}{n}\sum\lvert y_i-\hat y_i\rvert Default; robust to outliers
RMSE 1n∑(yi−y^i)2\sqrt{\frac{1}{n}\sum(y_i-\hat y_i)^2} Penalise large errors more
MAPE 100n∑∣yi−y^i∣/∣yi∣\frac{100}{n}\sum\lvert y_i-\hat y_i\rvert/\lvert y_i\rvert Cross-scale comparison; avoid near zero
sMAPE Bounded 0-200% variant of MAPE More stable near zero

API Quick Reference

from statsmodels.tsa.seasonal import seasonal_decompose
from statsmodels.tsa.stattools import adfuller
from statsmodels.tsa.statespace.sarimax import SARIMAX
from sklearn.model_selection import TimeSeriesSplit
import pmdarima as pm

decomp = seasonal_decompose(x, model='additive', period=7)
# Reject non-stationarity if p < 0.05
adf_stat, p_value, *_ = adfuller(x, autolag='AIC')

model = SARIMAX(endog, order=(p,d,q), seasonal_order=(P,D,Q,m))
fitted = model.fit(disp=False)
forecast = fitted.get_forecast(steps=28)

Decision Guide

Stationary (ADF p < 0.05)?
|-- No → difference it, re-test
`-- Yes
    |-- Single series, need uncertainty intervals?
    |   → ARIMA/SARIMA
    |-- Many related series / external features?
    |   → Lag-feature gradient boosting
    |-- Missing data / holiday effects? → Prophet
    `-- Very large dataset, many series?
        → Deep learning (LSTM / TFT)

AWS SageMaker Quick Reference

Task SageMaker approach
AutoML forecasting across many series SageMaker Canvas time-series forecasting (Amazon Forecast, create_auto_predictor, for existing accounts only)
Custom deep learning forecaster SageMaker built-in DeepAR algorithm
No-code forecasting SageMaker Canvas
online /books/classical-data-science/appendix-a-quick-reference Newcastle, UK v0.1.0 // rss