Appendix A
Quick Reference
Scan, don’t read. This appendix collects each chapter’s lookup material, organised by chapter: key terms, API summaries, formulas, decision guides, and metrics. Keep it open while working; it is not meant to be read start to finish.
Chapter 1 — History & Landscape
AI / ML Timeline at a Glance
| Year | Event | Why it matters |
|---|---|---|
| 1950 | Turing Test | An early operational test for machine intelligence |
| 1956 | Dartmouth workshop | Founds AI as a field (name coined in the 1955 proposal) |
| 1986 | Backpropagation popularised | Makes multi-layer networks trainable |
| 1995 | SVMs | Strong theoretical foundations for classification |
| 1997 | LSTM | Recurrent networks that learn long-range dependencies |
| 2001 | Random Forests | Still a strong tabular baseline 20+ years later |
| 2009 | ImageNet released | Large labelled dataset enables scale |
| 2012 | AlexNet | Deep learning goes mainstream |
| 2017 | Transformer (“Attention Is All You Need”) | Foundation of modern LLMs |
| 2018 | BERT | Transfer learning for NLP |
| 2020 | GPT-3 | Few-shot learning; scale changes the game |
| 2022 | ChatGPT | RLHF + instruction tuning → conversational AI |
The Three Eras
| Era | Rough dates | Core idea | Still used? |
|---|---|---|---|
| Rules & Expert Systems | 1950s-1980s | Hand-coded logic | Rarely, in very narrow domains |
| Statistical ML | 1980s-2010s | Learn patterns from labelled data | Yes — dominates tabular ML |
| Deep Learning / Foundation Models | 2012-now | Scale neural nets on massive data | Yes — dominates unstructured data |
Role Definitions
| Role | Primary tool | Spends most time on |
|---|---|---|
| Data Analyst | SQL, BI tools | Describing what happened |
| Data Engineer | Spark, dbt, Kafka | Moving and storing data |
| Data Scientist | sklearn, PyTorch | Building predictive models |
| ML Engineer | MLflow, Docker, k8s | Making models work in production |
| Research Scientist | Papers, JAX/PyTorch | Advancing the state of the art |
| AI/ML Solutions Architect | All of the above | Designing systems, advising customers |
Nested Scope of Terms
Artificial Intelligence
`-- Machine Learning
|-- Classical ML (linear models, trees, SVMs, ensembles)
`-- Deep Learning
|-- CNNs (images)
|-- RNNs / LSTMs (sequences)
`-- Transformers (text, code, images, multimodal)
`-- Foundation Models (pretrained, then adapted;
| text, vision, multimodal)
`-- Large Language Models (GPT, Claude, Llama, Gemini)
When to Use Which Approach
| Situation | Reach for |
|---|---|
| Tabular data, clear features, <1M rows | XGBoost / LightGBM / Random Forest |
| Tabular data, need to explain decisions | Logistic Regression + SHAP |
| Images, audio, video | CNN or Vision Transformer |
| Text: classification, extraction | Fine-tuned BERT or LLM |
| Text: generation, chat, summarisation | Foundation model (prompt or fine-tune) |
| Time series forecasting | Prophet / ARIMA / Temporal Fusion Transformer |
| Small dataset, simple relationship | Linear / logistic regression |
| Need to deploy cheaply | Classical ML almost always wins |
Essential pandas One-Liners
df.shape # (rows, cols)
df.dtypes # column types
df.isnull().sum() # missing values per column
df.describe() # summary stats (numeric)
df['col'].value_counts() # frequency of each category
df.groupby('col')['val'].mean() # grouped mean
df.sort_values('col', ascending=False) # sort descending
df[df['col'] > threshold] # filter rowsEssential matplotlib One-Liners
plt.figure(figsize=(12, 5)) # set figure size
plt.plot(x, y, label='name', color='#4E79A7') # line
plt.scatter(x, y, s=50, alpha=0.5) # scatter
plt.axvline(x=val, linestyle='--') # vertical reference line
plt.xlabel('X Label')
plt.ylabel('Y Label')
plt.title('Title')
plt.legend()
plt.tight_layout()
plt.savefig('file.png', dpi=150, bbox_inches='tight')
plt.show()Chapter 2 — Business Problem Framing
Key Terms
| Term | Definition |
|---|---|
| Label | The exact, measurable quantity being predicted, tied to a real event |
| Unit of analysis | What one row of training data represents |
| Prediction horizon | How far in advance of the event the prediction must be made |
| Data leakage | A feature that encodes information unavailable at prediction time |
ML Problem Type Taxonomy
| Type | Label | Algorithm family | Use when |
|---|---|---|---|
| Binary classification | 0 or 1 | Logistic regression, tree-based, neural net | Predict yes/no event |
| Multi-class classification | One of N classes | Same as binary + softmax | Predict one category from several |
| Multi-label classification | Multiple 0/1 flags | Sigmoid per class | One item can belong to multiple categories |
| Regression | Continuous number | Linear regression, tree-based, neural net | Predict a quantity |
| Ranking | Relative order | LTR models, collaborative filtering | Order items by relevance per user |
| Clustering | None (unsupervised) | K-means, DBSCAN, hierarchical | Discover natural groups |
| Anomaly detection | Rare or unlabelled | Isolation Forest, Autoencoder, statistical | Find unusual examples |
| Time series forecasting | Future value | ARIMA, Prophet, TFT | Predict next N values in a sequence |
| Sequence labelling | Tag per token | CRF, LSTM, BERT | NER, POS tagging |
Problem Statement Template
GOAL STATEMENT
Purpose: what this project is trying to do, in plain language
Advantage: the business benefit, tied to money or a tracked metric
Measurement: how success is verified, stated as a number
ML PROBLEM DEFINITION
Label: the exact, measurable event being predicted
Unit of analysis: what one training row represents
Features available: confirmed available at prediction time, not after
Prediction horizon: how far ahead of the event the prediction is made
Is ML the right tool? checked against the Suitability Checklist below
SUCCESS METRIC
Business outcome: ________________________
Target number: ________________________
A blank field is the signal Procedure 2.1 warns about: the framing isn’t finished yet.
The ML Suitability Checklist
[ ] There is a clear, measurable prediction target
[ ] Historical data exists (or can be constructed)
[ ] Features/label relationship is stable over time
[ ] Enough examples exist (rule of thumb: >= 1,000/class)
[ ] Features at prediction time are enough to predict
[ ] A simple rule does NOT already work well enough
[ ] The cost-benefit of ML vs. rule justifies the effort
[ ] The action enabled by the prediction is operationalised
AWS SageMaker Quick Reference
| Task | SageMaker approach | Documentation |
|---|---|---|
| Share a structured dataset across a team | Upload to S3, register in SageMaker Feature Store | sagemaker.feature_store |
Chapter 3 — Maths Foundations: Statistics, Probability & Linear Algebra
Key Terms
| Term | Definition |
|---|---|
| Prior | : belief about a hypothesis before seeing evidence |
| Likelihood | : probability of the evidence given the hypothesis |
| Posterior | : updated belief after accounting for evidence |
| Variance | Average squared deviation from the mean, |
| Correlation (Pearson r) | Scale-independent measure of linear association, |
| p-value | Probability, under , of data at least as extreme as what was observed |
| Effect size | A measure (e.g. Cohen’s ) of how large a difference is, independent of sample size |
| Normal equations | The closed-form least-squares solution |
Probability Distributions Cheat Sheet
| Distribution | Parameters | Mean | Variance | scipy / numpy | Use when |
|---|---|---|---|---|---|
| Gaussian | stats.norm(mu, sigma) |
Continuous, symmetric quantities | |||
| Bernoulli | stats.bernoulli(p) |
Single binary outcome | |||
| Binomial | stats.binom(n, p) |
Count of successes in trials | |||
| Poisson | stats.poisson(lam) |
Count of rare events in a fixed interval | |||
| Uniform | np.random.uniform(a,b) |
Equal probability over a range | |||
| Exponential | stats.expon(scale=1/lam) |
Time between events | |||
| Beta | stats.beta(a, b) |
Probability of a probability (Bayesian) |
Bayes’ Theorem
In ML: = observed features (evidence), = the label (hypothesis). is the prior (base rate); is the likelihood; is the posterior: the quantity a classifier is ultimately trying to produce.
Descriptive Statistics Summary
| Statistic | Formula | numpy | Notes |
|---|---|---|---|
| Mean | np.mean(x) |
Sensitive to outliers | |
| Median | Middle value | np.median(x) |
Robust to outliers |
| Variance | np.var(x) |
Use ddof=1 for sample variance |
|
| Std dev | np.std(x) |
Same units as | |
| Covariance | np.cov(x,y) |
Not normalised; scale-dependent. np.cov divides by by default |
|
| Correlation | / | np.corrcoef(x,y) |
Scale-independent, |
| IQR | np.percentile(x,[25,75]) |
Robust spread measure |
Common Statistical Tests
| Test | Use when | Assumes | scipy call |
|---|---|---|---|
| Shapiro-Wilk | Test if data is Gaussian () | — | stats.shapiro(x) |
| t-test (independent) | Compare means of 2 groups | Gaussian, equal variance | stats.ttest_ind(a, b) |
| Welch’s t-test | Compare means of 2 groups (unequal variance) | Gaussian | stats.ttest_ind(a, b, equal_var=False) |
| Mann-Whitney U | Compare 2 groups (non-parametric) | None | stats.mannwhitneyu(a, b) |
| ANOVA | Compare means of 3+ groups | Gaussian, equal variance | stats.f_oneway(*groups) |
| Chi-squared | Association between 2 categorical variables | Expected counts | stats.chi2_contingency(table) |
| Pearson r | Linear correlation | Linear relationship; Gaussian only for the p-value | stats.pearsonr(x, y) |
| Spearman | Monotonic correlation (non-parametric) | None | stats.spearmanr(x, y) |
Linear Algebra Operations
| Operation | Notation | numpy | Shape |
|---|---|---|---|
| Vector dot product | a @ b |
scalar | |
| Matrix multiply | A @ B |
||
| Transpose | A.T |
||
| Inverse | np.linalg.inv(A) |
||
| Solve | np.linalg.solve(A, b) |
More stable than inv |
|
| Eigendecomposition | np.linalg.eig(A) |
Returns (eigenvalues, eigenvectors) | |
| SVD | np.linalg.svd(A) |
Returns (U, s, Vt) | |
| L2 norm | np.linalg.norm(v) |
→ scalar | |
| L1 norm | np.linalg.norm(v, ord=1) |
→ scalar | |
| Element-wise multiply | A * B |
Same shape as |
Normal Equations for Linear Regression
X_bias = np.column_stack([np.ones(len(X)), X])
w = np.linalg.solve(X_bias.T @ X_bias, X_bias.T @ y)
# w[0] = intercept, w[1:] = feature weightsLimitations: complexity in the number of features , slow for large . Fails if is singular (collinear features). For collinear features, use Ridge (Chapter 18) or a least-squares solver such as np.linalg.lstsq. For very large , use gradient descent (Chapter 4).
Correlation Interpretation Guide
| range | Strength |
|---|---|
| 0.9 – 1.0 | Very strong |
| 0.7 – 0.9 | Strong |
| 0.5 – 0.7 | Moderate |
| 0.3 – 0.5 | Weak |
| 0.0 – 0.3 | Negligible |
Caveats: Pearson measures linear correlation only; does not imply independence (could be non-linear); correlation is not causation; Spearman is more robust to outliers and handles monotonic non-linear relationships.
AWS SageMaker Quick Reference
| Task | SageMaker approach | Documentation |
|---|---|---|
| Statistical analysis via UI, no code | SageMaker Data Wrangler | sagemaker.wrangler |
| Statistics on datasets too large for local memory (roughly GB) | SageMaker Processing Jobs (ScriptProcessor) |
sagemaker.processing.ScriptProcessor |
Chapter 4 — Maths Foundations: Calculus & Optimisation
Key Terms
| Term | Definition |
|---|---|
| Derivative | Rate of change of a function at a point, |
| Gradient | Vector of partial derivatives, ; points in the direction of steepest ascent |
| Learning rate () | Step size in the gradient descent update rule |
| Backpropagation | Repeated application of the chain rule to compute gradients through a composed (layered) function |
| Convex loss | A loss surface where every local minimum is a global minimum; gradient descent reaches it with a small enough learning rate |
| Hessian | Matrix of second derivatives; used by second-order optimisers, for parameters |
Gradient Descent Update Rule
Gradient Descent Variants
| Variant | Batch size | numpy pattern | Pros | Cons |
|---|---|---|---|---|
| Batch GD | All examples | X.T @ (X@w - y) |
Exact gradient | Slow for large |
| Stochastic GD (SGD) | 1 example | X[i] @ (X[i]@w - y[i]) |
Fast; noise can escape local minima | Very noisy, hard to converge |
| Mini-batch GD | examples (32-256) | X[batch].T @ (X[batch]@w - y[batch]) |
Best of both; standard in practice | Requires batch-size tuning |
Common Loss Functions
| Loss | Formula | Use for | Gradient |
|---|---|---|---|
| MSE | Regression | ||
| MAE | Regression, outlier-robust | ||
| Huber | MSE if , else MAE | Regression, robust | Hybrid of MSE/MAE gradients |
| Binary cross-entropy | Binary classification | with respect to the logit, with sigmoid output | |
| Categorical cross-entropy | Multi-class classification | with respect to the logits, with softmax output | |
| Hinge | SVM | if , else |
Optimiser Comparison
| Optimiser | Update rule | Learning-rate sensitivity | Best for |
|---|---|---|---|
| SGD | High | Simple models, full manual control | |
| SGD + momentum | ; | Moderate | Deep learning with careful tuning |
| Adam | Adaptive per-parameter learning rate | Low | Default for neural networks and fine-tuning |
| AdamW | Adam + decoupled weight decay | Low | Transformer training (standard) |
| L-BFGS | Quasi-Newton (2nd-order) | Very low | Smooth, full-batch problems (classical ML); memory grows linearly with parameter count |
| RMSprop | Adaptive learning rate, no momentum | Moderate | RNNs, non-stationary objectives |
Adam Algorithm (Simplified)
alpha, beta1, beta2, eps = 0.001, 0.9, 0.999, 1e-8
m, v = np.zeros_like(w), np.zeros_like(w)
for t in range(1, n_steps + 1):
g = gradient(w)
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * g**2
m_hat = m / (1 - beta1**t)
v_hat = v / (1 - beta2**t)
w = w - alpha * m_hat / (np.sqrt(v_hat) + eps)Learning Rate Schedule Types
| Schedule | Behaviour | When to use |
|---|---|---|
| Constant | Fixed throughout training | Simple baseline |
| Step decay | Halve every epochs | Traditional CNN training |
| Cosine decay | Smooth decay to 0 | General neural network training |
| Warm-up + cosine | Linear ramp then cosine decay | Transformer training (standard) |
| Reduce on plateau | Reduce when validation loss stalls | When unsure of a fixed schedule |
| Cyclic | Oscillates between min/max | Helps explore the loss landscape |
Key Calculus Facts for ML
| Concept | Formula | ML context |
|---|---|---|
| Derivative of | MSE gradient | |
| Derivative of | Cross-entropy gradient | |
| Derivative of | Sigmoid layer backprop | |
| Derivative of | if else | ReLU layer backprop |
| Chain rule | Backpropagation | |
| Product rule | Occasionally needed |
scipy.optimize Cheat Sheet
import scipy.optimize as opt
# Minimise scalar fn
result = opt.minimize(fun, x0, method='L-BFGS-B', jac=gradient_fn)
# Find f(x) = 0
result = opt.root(fun, x0)
# 1D, no gradient needed
result = opt.minimize_scalar(fun, bounds=(a, b), method='bounded')
# Least squares fit
popt, pcov = opt.curve_fit(
model_fn, xdata, ydata, p0=initial_params
)
result.x # optimal parameters
result.fun # final function value (loss)
result.success # bool: did it converge?
result.nit # number of iterations| Method | Best for | Notes |
|---|---|---|
'L-BFGS-B' |
General smooth optimisation | Usually best default |
'Nelder-Mead' |
No gradient available | Slow but robust |
'SLSQP' |
Constrained optimisation | Handles inequality constraints |
'Newton-CG' |
Smooth, exact gradient available | Fast convergence |
Diagnosing Training Problems
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss goes to NaN | LR too high, or numerical overflow | Reduce LR 10x; add gradient clipping |
| Loss oscillates | LR too high | Reduce LR 3-10x |
| Loss plateaus early | LR too low, or converged to local min | Increase LR; use momentum/Adam |
| Val loss increases while train decreases | Overfitting | Regularisation, more data |
| Both train and val loss plateau high | Underfitting | Larger model, more epochs, better features |
| Loss decreases but very slowly | LR too low, or vanishing gradients | LR schedule; batch norm; residual connections |
AWS SageMaker Quick Reference
| Task | SageMaker approach | Documentation |
|---|---|---|
| Set training hyperparameters (learning rate, epochs, batch size) | sagemaker.estimator.Estimator(hyperparameters={...}) |
sagemaker.estimator |
| Search over learning rate / hyperparameters automatically | SageMaker Automatic Model Tuning (Bayesian optimisation) | Covered in a later chapter on hyperparameter tuning |
Chapter 5 — Information Theory Basics
Key Terms
| Term | Definition |
|---|---|
| Self-information | : surprise of a single event with probability |
| Entropy | Average information content of a distribution; measures uncertainty |
| Cross-entropy | Average cost of using predicted distribution when the truth is |
| KL divergence | Directed “distance” of from ; always , asymmetric |
| Mutual information | How much knowing reduces uncertainty about (and vice versa); zero iff independent |
| Nat vs. bit | Unit of information: natural log gives nats, gives bits; |
Key Formulas
| Name | Formula | When you use it |
|---|---|---|
| Self-information | Building block for everything below | |
| Entropy | Node impurity (trees), dataset uncertainty | |
| Cross-entropy | Classification loss function | |
| Binary cross-entropy | Binary classifier loss | |
| KL divergence | VAE regularisation, RLHF penalty, drift detection | |
| Identity | Explains why minimising cross-entropy minimises KL | |
| Mutual information | Feature selection, representation learning |
API / Function Summary
scipy.stats.entropy(pk, qk=None, base=None, axis=0): returns entropy of pk, or if qk is given. pk need not be pre-normalised. base=None gives nats; base=2 gives bits.
h = scipy.stats.entropy([0.5, 0.5], base=2) # 1.0 bit
kl = scipy.stats.entropy(p, q, base=2) # KL(p || q) in bitslog_loss, called as sklearn.metrics.log_loss(y_true, y_pred, labels=None, normalize=True), returns the cross-entropy (log) loss. y_pred is predicted probabilities, shape (n_samples,) or (n_samples, n_classes).
loss = log_loss(y_true, y_pred_proba)sklearn.feature_selection.mutual_info_classif(X, y, discrete_features='auto', n_neighbors=3, random_state=None) (and mutual_info_regression for continuous targets): returns an ndarray of shape (n_features,), estimated mutual information with the target in nats, non-negative.
mi = mutual_info_classif(X, y, random_state=42)
ranking = (
pd.Series(mi, index=X.columns).sort_values(ascending=False)
)Algorithm / Approach Comparison
| Approach | Best for | Weakness | Function |
|---|---|---|---|
| Correlation (Pearson) | Quick linear-relationship screening | Misses non-linear and non-monotonic relationships entirely | df.corr(), np.corrcoef |
| Mutual information | Any relationship, linear or not | Needs more data to estimate reliably; no fixed scale for “how strong” | mutual_info_classif / _regression |
| Cross-entropy / log loss | Classifier training loss and evaluation | Only defined for probabilistic predictions, sensitive to miscalibration | log_loss, framework loss functions |
| KL divergence | Comparing two distributions directly (drift, VAE, RLHF) | Asymmetric, undefined if support doesn’t match | scipy.stats.entropy(p, q) |
Decision Guide
Do you need to rank features by predictive power?
|-- Yes
| |-- Only linear relationships, want speed?
| | → correlation
| `-- Non-linear relationships, or unsure?
| → mutual_info_classif/regression
`-- No
|-- Training/evaluating a classifier?
| → cross-entropy / log_loss
`-- Comparing two known distributions
(drift, prior vs. posterior)? → KL divergence
Metrics at a Glance
| Metric | Range | Higher/lower is “better”? | Use when |
|---|---|---|---|
| Entropy | to | Context-dependent (not a “quality” metric) | Measuring how uncertain a distribution is |
| Cross-entropy | to | Lower is better | Classifier training loss |
| KL divergence | to | Lower (closer to 0) means more similar | Distribution comparison, drift detection |
| Mutual information | to the smaller of and (discrete variables) | Higher means more informative | Feature selection ranking |
Numerical Pitfalls Cheat Sheet
| Pitfall | Symptom | Fix |
|---|---|---|
nan or -inf loss |
Clip probabilities to , e.g. | |
| Unnormalised distribution | Entropy/KL values don’t match expectations | Divide by .sum() before computing, or let scipy.stats.entropy normalise for you |
| has a zero where doesn’t | Smooth with a small epsilon, or use a KL variant with support handling | |
| Comparing MI scores across different datasets | Misleading “this feature is better” conclusions | Only rank features within the same run/dataset |
AWS SageMaker Quick Reference
| Task | SageMaker approach | Documentation |
|---|---|---|
| Mutual-information feature selection at scale | SKLearnProcessor running a script that calls mutual_info_classif |
SageMaker Processing SDK |
| Monitoring prediction distribution drift | SageMaker Model Monitor (built-in data quality / drift checks) | SageMaker Model Monitor docs |
| Cross-entropy training loss on GPU | Any Estimator (PyTorch/TensorFlow): loss defined in your training script |
SageMaker Training Jobs |
Chapter 6 — Python Data Science Ecosystem
Key Terms
| Term | Definition |
|---|---|
| Broadcasting | NumPy applying an operation across arrays of compatible shapes without an explicit loop |
| Series | A 1D labelled array: a single pandas column |
| DataFrame | A 2D labelled table, potentially with different types per column |
| Index | The row-label structure that lets pandas align data across sources |
| Estimator | A scikit-learn object (model or transformer) that follows the .fit()/.predict()/.transform() API |
Library Purpose Table
| Library | Role | When you need it |
|---|---|---|
numpy |
Fast N-dimensional arrays, maths | Any numerical computation, linear algebra |
pandas |
Labelled tabular data | Loading, cleaning, transforming, grouping datasets |
matplotlib |
Low-level plotting | Custom charts, fine-grained control |
seaborn |
Statistical visualisation | EDA, distributions, categorical comparisons |
scipy |
Scientific computing | Statistical tests, optimisation, signal processing |
scikit-learn |
ML algorithms + tooling | Models, preprocessing, cross-validation, metrics |
jupyter |
Interactive notebooks | Exploration, prototyping, sharing results |
NumPy Quick Reference
np.array([1, 2, 3]); np.zeros((3, 4)); np.ones((2, 3)); np.eye(4)
np.arange(start, stop, step); np.linspace(start, stop, n)
np.random.randn(n); np.random.randint(low, high, size)
a.shape; a.dtype; a.ndim; a.size
a + b, a - b, a * b, a / b # element-wise (broadcasting)
a @ b # matrix multiply
a.T # transpose
np.sum(a, axis=0) # sum along axis
np.mean(a), np.std(a), np.var(a)
a[a > threshold] # boolean indexing
np.where(condition, x, y) # conditional selection
a.reshape(3, 4); a.flatten(); a.ravel()pandas Quick Reference
pd.read_csv('file.csv'); pd.read_parquet('file.parquet')
df.shape; df.dtypes; df.head(n); df.info(); df.describe()
df.isnull().sum(); df.nunique()
df['col']; df[['col_a', 'col_b']]
df.loc[label]; df.iloc[n]
df[df['col'] > value]; df.query("col > @value")
df['new'] = df['a'] + df['b']
df.rename(columns={'old': 'new'}); df.drop(columns=['col'])
df.sort_values('col', ascending=False)
df.drop_duplicates(subset=['col'])
df.fillna(value); df.dropna(subset=['col'])
df.groupby('col')['val'].mean()
df.groupby(['a', 'b'])['val'].agg(['mean', 'count', 'std'])
df.pivot_table(values='v', index='r', columns='c', aggfunc='mean')matplotlib / seaborn Quick Reference
fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(x, y, color='#4E79A7', linewidth=2, label='name')
ax.scatter(x, y, s=50, alpha=0.5, c=colours, cmap='viridis')
ax.bar(categories, values, color='steelblue', edgecolor='white')
ax.set_title('Title'); ax.set_xlabel('X'); ax.set_ylabel('Y')
ax.legend()
plt.savefig('out.png', dpi=150, bbox_inches='tight')
sns.histplot(data=df, x='col', hue='group', kde=True, bins=30)
sns.boxplot(data=df, x='cat', y='val', hue='group')
sns.barplot(data=df, x='cat', y='val', errorbar='sd')
sns.scatterplot(data=df, x='a', y='b', hue='label', alpha=0.5)
sns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm', center=0)
sns.pairplot(df, hue='target')scikit-learn API Pattern
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
score = model.score(X_test, y_test)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train) # fit on train
X_test_s = scaler.transform(X_test) # apply to test (no fit!)
from sklearn.pipeline import Pipeline
pipe = Pipeline([
('scale', StandardScaler()),
('model', LogisticRegression()),
])
pipe.fit(X_train, y_train)Common Imports Block
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from sklearn.model_selection import (
train_test_split,
cross_val_score,
)
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.metrics import accuracy_score, classification_report
from sklearn.pipeline import Pipeline
plt.rcParams['figure.figsize'] = (10, 5)
sns.set_style('whitegrid')
np.random.seed(42)Troubleshooting Quick Reference
| Error / symptom | Most likely cause | Fix |
|---|---|---|
ModuleNotFoundError: No module named 'numpy' |
Wrong environment active | Activate the right env, then reinstall |
ValueError: Input contains NaN |
sklearn received missing values | Clean data first (Chapter 7) |
DataConversionWarning: A column-vector y was passed |
y is shape (n,1) not (n,) |
y.ravel() or y.squeeze() |
| Plot is blank / not showing | plt.show() before data calls |
Move data calls above plt.show() |
seaborn hue not working |
Column has NaN values | df.dropna(subset=['hue_col']) first |
Chapter 7 — Data Cleaning & Wrangling
Key Terms
| Term | Definition |
|---|---|
| MCAR | Missing Completely At Random: missingness unrelated to any variable, observed or not |
| MAR | Missing At Random: missingness depends on observed variables, not the missing value |
| MNAR | Missing Not At Random: missingness depends on the missing value. Naive imputation biases results |
| Winsorising | Capping values beyond a percentile threshold at that threshold, rather than removing them |
| IQR | Interquartile range, ; used to define robust, skew-resistant outlier bounds |
Data Quality Problem Types
| Problem | Detection | Treatment |
|---|---|---|
| Missing values | df.isnull().sum() |
Impute, drop, or flag |
| Outliers | IQR, z-score, domain knowledge | Cap, remove, transform, or flag |
| Duplicates | df.duplicated().sum() |
drop_duplicates() |
| Wrong types | df.dtypes |
pd.to_numeric(), pd.to_datetime(), .astype() |
| Inconsistent categories | .value_counts() |
.str.strip().str.lower() + .map() |
| Encoding errors | Scan .value_counts() for a repeated Ã-plus-character pattern |
On rows containing à only: .str.encode('latin1').str.decode('utf-8') (reverses a UTF-8-read-as-Latin-1 mismatch) |
| Schema violations | Rule-based checks (Procedure 7.6), not just .dtypes |
Drop, correct, or impute, once the rule that was broken is known |
Missing Value Imputation Comparison
| Strategy | When to use | Risk |
|---|---|---|
| Drop rows | < 5% missing, MCAR | Loses data |
| Mean fill | Numeric, symmetric | Sensitive to outliers |
| Median fill | Numeric, skewed | Slightly less precise than mean for normal data |
| Mode fill | Categorical | Can over-represent majority class |
| Forward fill | Time series | Wrong if gap is long |
| KNN imputation | Other features correlate with the missing one | Slow on large datasets |
| Model-based | MAR, with strong predictors of the missing column | Added complexity; risk of leakage if fit on the full dataset |
Outlier Detection Methods
| Method | Formula | Best for |
|---|---|---|
| IQR | Outside or | Skewed distributions, robust |
| Z-score | , | Roughly normal distributions |
| Percentile | Below or above | Any distribution, simple |
| Domain cap | Hard limits from business rules | When the valid range is known a priori |
| Isolation Forest | Tree-based anomaly score | Multivariate, complex patterns (Chapter 19) |
dtype Conversion Table
| From | To | Method |
|---|---|---|
| Text (string number) | float64 |
pd.to_numeric(col, errors='coerce') |
| Text (string number) | Int64 (nullable int) |
pd.to_numeric(col).astype('Int64') |
| Text (date string) | datetime64 |
pd.to_datetime(col, errors='coerce') |
float64 |
int64 (after removing NaN) |
.dropna().astype(int) |
| Text | category |
.astype('category') |
int (0/1) |
bool |
.astype(bool) |
datetime64 |
Text (formatted string) | .dt.strftime('%Y-%m-%d') |
Cleaning Checklist
[ ] 1. Check shape (rows x cols)
[ ] 2. Check dtypes: fix wrong types immediately
[ ] 3. Count missing values per column
[ ] 4. Count and remove exact duplicates
[ ] 5. Inspect category columns: normalise strings
[ ] 6. Detect numeric outliers (IQR or z-score)
[ ] 7. Decide imputation strategy per column
[ ] 8. Apply imputation
[ ] 9. Validate: assert on domain rules
[ ] 10. Compare before/after distributions visually
Troubleshooting Common Errors
| Error / symptom | Most likely cause | Fix |
|---|---|---|
ValueError: Input contains NaN (sklearn) |
Missing values not imputed | Check df.isnull().sum(), impute first |
| Category encoding produces 20+ categories | Inconsistent string values | Normalise with .str.lower().str.strip() + map |
| KNN imputer runs very slowly | Too many columns included | Include only numeric columns relevant to the target |
| Duplicates reappear after a merge | Join key not unique on one side | Check key uniqueness first, or pass validate='one_to_one' to merge() |
pd.to_datetime returns all NaT |
Unexpected date format | Specify format= explicitly, e.g. format='%d/%m/%Y' |
This chapter has no AWS SageMaker Quick Reference. Chapter 7’s cleaning operations are pandas/scikit-learn work performed before any AWS-side step. The SageMaker Feature Store material in Appendix A’s Chapter 2 entry is where a cleaned dataset like this chapter’s next gets registered.
Chapter 8 — SQL for Data Science
Key Terms
| Term | Definition |
|---|---|
| Common table expression (CTE) | A named subquery defined with WITH, used for readability, not performance |
| Window function | A function computed across rows related to the current row, without collapsing them the way GROUP BY does |
| Partition | The PARTITION BY grouping a window function restarts its calculation within, analogous to GROUP BY but non-collapsing |
| Fact table | The table of events/transactions being enriched (e.g. orders) |
| Dimension table | The table of descriptive attributes joined onto a fact table (e.g. customers, products) |
Window Function Syntax
function_name() OVER (
PARTITION BY col1, col2 -- restart calculation for each group
ORDER BY col3 -- row order within the partition
ROWS BETWEEN n PRECEDING -- optional: window frame
AND CURRENT ROW
)Window Function Quick Reference
| Function | Returns | Example |
|---|---|---|
ROW_NUMBER() |
Unique sequential integer (no ties) | ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC) |
RANK() |
Rank with gaps on ties (1,1,3) | RANK() OVER (ORDER BY score DESC) |
DENSE_RANK() |
Rank without gaps (1,1,2) | DENSE_RANK() OVER (ORDER BY score DESC) |
NTILE(n) |
Bucket into groups (quartiles etc.) | NTILE(4) OVER (ORDER BY salary) |
LAG(col, n) |
Value of col rows back |
LAG(revenue, 1) OVER (ORDER BY date) |
LEAD(col, n) |
Value of col rows ahead |
LEAD(revenue, 1) OVER (ORDER BY date) |
FIRST_VALUE(col) |
First value in partition | FIRST_VALUE(order_date) OVER (PARTITION BY customer_id ORDER BY order_date) |
LAST_VALUE(col) |
Last value in partition | Requires ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING |
SUM(col) OVER (...) |
Running total (add ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW if dates can tie) |
SUM(rev) OVER (ORDER BY date) |
AVG(col) OVER (...) |
Moving / partition average | AVG(rev) OVER (PARTITION BY dept) |
COUNT(*) OVER () |
Grand total row count | Same value on every row |
Aggregation Patterns
-- Basic group aggregation
SELECT col, COUNT(*), SUM(val), AVG(val), MIN(val), MAX(val)
FROM table GROUP BY col ORDER BY SUM(val) DESC;
-- Filter groups with HAVING
GROUP BY col HAVING SUM(val) > 1000
-- Multiple grouping levels
GROUP BY ROLLUP(region, dept) -- subtotals at each level
-- Conditional aggregation
SELECT
SUM(CASE WHEN status = 'completed'
THEN revenue ELSE 0 END) AS completed_rev,
SUM(CASE WHEN status = 'returned'
THEN revenue ELSE 0 END) AS returned_rev
FROM orders;CTE Template
WITH
cte_name_1 AS ( SELECT ... ),
cte_name_2 AS ( SELECT ... FROM cte_name_1 WHERE ... ),
cte_name_3 AS (
SELECT
*,
RANK() OVER (PARTITION BY ... ORDER BY ...) AS rank
FROM cte_name_2
)
SELECT * FROM cte_name_3 WHERE rank <= 10;Join Types
| Join | Rows kept |
|---|---|
INNER JOIN |
Only rows matching in both tables |
LEFT JOIN |
All rows from left, matched rows from right (NULLs where no match): the dominant join in analytics |
RIGHT JOIN |
Mirror of LEFT JOIN (rare: reorder tables and use LEFT JOIN instead) |
FULL OUTER JOIN |
All rows from both tables, NULLs where no match on either side |
CROSS JOIN |
Every combination of left × right rows (use carefully) |
-- Most common analytical join pattern
FROM fact_table f
LEFT JOIN dim_table d ON f.foreign_key = d.primary_keyDate Functions (DuckDB / ANSI SQL)
DATE_TRUNC('month', date_col) -- first day of month
DATE_TRUNC('year', date_col) -- first day of year
DATE_PART('month', date_col) -- extract month number
DATEDIFF('day', start_date, end_date) -- days between dates
date_col + INTERVAL '30 days' -- add 30 days
date_col + INTERVAL '1 month' -- add 1 month
CURRENT_DATE -- todayDuckDB vs sqlite3 vs pandas.query
| Feature | DuckDB | sqlite3 | pandas.query |
|---|---|---|---|
| Window functions | Full support | Yes (SQLite 3.25+) | No |
| CTEs | Yes | Yes | No |
| Multi-table JOINs | Yes | Yes | No |
| Reads pandas DataFrames | Direct | Via manual insert | N/A: IS pandas |
| Parallelism | Multi-core | Single-threaded | Partial (numpy) |
| Aggregation on millions of rows | Fast | Slow | Slow |
| Setup | pip install duckdb |
stdlib | Part of pandas |
Common Analytical Query Skeletons
-- Top-N per group
WITH ranked AS (
SELECT
*,
RANK() OVER (
PARTITION BY group_col ORDER BY metric DESC
) AS r
FROM table
)
SELECT * FROM ranked WHERE r <= 3;
-- Period-over-period comparison
WITH monthly AS (
SELECT
DATE_TRUNC('month', date_col) AS m,
SUM(val) AS rev
FROM t
GROUP BY 1
)
SELECT
m,
rev,
LAG(rev) OVER (ORDER BY m) AS prev,
(rev / LAG(rev) OVER (ORDER BY m) - 1) * 100 AS pct
FROM monthly;
-- Running total
SELECT
date_col,
val,
SUM(val) OVER (
ORDER BY date_col
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS running_total
FROM t;
-- Deduplicate (keep latest per ID)
WITH ranked AS (
SELECT
*,
ROW_NUMBER() OVER (
PARTITION BY id ORDER BY updated_at DESC
) AS r
FROM t
)
SELECT * FROM ranked WHERE r = 1;SQL vs pandas Decision Guide
| Use SQL when | Use pandas when |
|---|---|
| Data already lives in a database or warehouse | Data is already loaded as a DataFrame |
| Query joins more than two tables | The transformation is custom, specific to feature engineering |
| Aggregation runs over rows too numerous to hold in memory | Task is quick exploration of a CSV never destined for a database |
AWS Athena / Redshift Quick Reference
| Task | AWS approach | Notes |
|---|---|---|
| Query Parquet/CSV directly from S3 | Amazon Athena (serverless SQL) | CREATE EXTERNAL TABLE ... LOCATION 's3://...' |
| Data warehouse queries at scale | Amazon Redshift | Window function syntax very close to DuckDB’s |
| Prefer over local pandas + DuckDB | Once data lives in S3 and is too large for memory | Billed per byte of data scanned |
Chapter 9 — Exploratory Data Analysis & Visualisation
Key Terms
| Term | Definition |
|---|---|
| Univariate analysis | Examining one variable in isolation (distribution, spread, outliers) |
| Bivariate analysis | Examining the relationship between two variables (e.g. feature vs. target) |
| Multivariate analysis | Examining relationships among three or more variables at once |
| Skewness | Asymmetry of a distribution; positive = long right tail, negative = long left tail |
| Kurtosis | “Tailedness” of a distribution; high kurtosis = more extreme outliers than a normal distribution |
| Cardinality | Number of unique values in a categorical column |
| Data snooping | Drawing conclusions from EDA that implicitly overfit to the full dataset, including the eventual test set |
Dataframe Profiling Cheat Sheet
df.shape # (rows, columns)
df.dtypes # column types
df.info() # types + non-null counts + memory
# Numeric summary: count, mean, std, min, 25/50/75%, max
df.describe()
# Categorical summary: count, unique, top, freq
df.describe(include='object')
df.isnull().sum() # missing count per column
df.duplicated().sum() # duplicate row count
df.nunique() # unique value count per column
# Random rows (better than .head() for spotting surprises)
df.sample(5)
df.select_dtypes(include='number') # numeric columns only
df.select_dtypes(include='object') # categorical columns onlyPlot Type Decision Guide
What are you looking at?
|-- One numeric variable
| |-- Shape/skew/modality?
| | -> histogram + KDE (sns.histplot(kde=True))
| |-- Quartiles/outliers only? -> boxplot
| `-- Shape AND quartiles? -> violin plot
|-- One categorical variable
| `-- Frequency of each level?
| -> bar chart (value_counts().plot(kind='bar'))
|-- Two numeric variables
| |-- Relationship shape? -> scatter plot
| `-- Strength across many pairs?
| -> correlation heatmap (sns.heatmap(df.corr()))
|-- One numeric + one categorical
| `-- Distribution shifts by group?
| -> boxplot/violin split by category, or KDE
|-- Two categorical variables
| `-- Association?
| -> cross-tab + stacked bar, or pivot heatmap
`-- 3+ numeric variables
`-- Full pairwise structure?
-> pairplot (sns.pairplot, hue=target)
Plot Type Comparison
| Plot | Shows | Best for | Weakness |
|---|---|---|---|
| Histogram | Frequency by value bucket | Distribution shape, skew, modality | Bin width choice changes the story |
| KDE plot | Smoothed density estimate | Comparing shapes of 2+ groups | Can smooth away real bimodality if bandwidth too wide |
| Boxplot | Median, quartiles, outliers | Quick outlier/spread comparison across many groups | Hides bimodality entirely |
| Violin plot | Boxplot + KDE combined | Comparing full shape across groups | Misleading with very small groups (n < ~20) |
| Scatter plot | Raw relationship between 2 numeric vars | Spotting non-linear relationships, clusters | Overplotting with large N (use alpha= or sample) |
| Correlation heatmap | Pearson/Spearman r for all pairs | Scanning many features for redundancy at once | Linear (Pearson) misses non-linear relationships |
| Pairplot | Scatter + histogram grid for all pairs | Full-dataset overview in one figure | Slow / unreadable beyond ~6 columns |
| Bar chart | Count or aggregate by category | Categorical frequency, group comparisons | Misleading if categories aren’t sorted or truncated |
Key Summary Statistics
| Statistic | Formula / call | What it tells you |
|---|---|---|
| Mean | series.mean() |
Central tendency, sensitive to outliers |
| Median | series.median() |
Central tendency, robust to outliers |
| Mode | series.mode() |
Most frequent value (numeric or categorical) |
| Std. deviation | series.std() |
Typical spread around the mean |
| IQR | Robust spread (middle 50% of data) | |
| Skewness | series.skew() |
right-skewed, left-skewed, symmetric |
| Kurtosis | series.kurt() |
heavier tails than normal, lighter |
| Pearson | series_a.corr(series_b) |
Linear relationship strength, to |
| Spearman | series_a.corr(series_b, method='spearman') |
Monotonic (linear or not) relationship strength |
The IQR-rule outlier bounds are:
The z-score outlier rule flags any value where:
Here is the population standard deviation (ddof=0), which is what scipy.stats.zscore uses. With values, no can exceed , so the rule can never fire when .
Outlier Detection Methods
| Method | Assumes | sklearn / scipy |
|---|---|---|
| IQR rule | Nothing about distribution shape | Manual (pandas) |
| Z-score | Roughly normal distribution | scipy.stats.zscore |
| Isolation Forest | Multivariate anomalies | sklearn.ensemble.IsolationForest |
| Local Outlier Factor | Local density variation matters | sklearn.neighbors.LocalOutlierFactor |
Common Imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
pd.set_option('display.max_columns', 50)
sns.set_style('whitegrid')
plt.rcParams['figure.figsize'] = (10, 5)AWS SageMaker Quick Reference
| Task | SageMaker approach | Notes |
|---|---|---|
| Automated data quality report | SageMaker Data Wrangler -> “Data Quality and Insights Report” | Flags missing values, imbalance, target leakage, outliers automatically |
| Interactive EDA at scale | SageMaker Studio notebook + pandas/seaborn on data from S3/Athena | Same code as local, larger instance for bigger data |
| Distributed profiling for very large datasets | SageMaker Processing job running a Spark/Deequ profiling script | Needed once data exceeds single-notebook memory |
| Feature-target correlation / leakage detection | Data Wrangler “Quick Model” analysis | Trains a throwaway model to estimate feature importance during EDA |
Chapter 10 — The Data Science Lifecycle
Key Terms
| Term | Definition |
|---|---|
| CRISP-DM | Cross-Industry Standard Process for Data Mining: business understanding, data understanding, data preparation, modelling, evaluation, deployment |
| TDSP | Microsoft’s Team Data Science Process: adds explicit deployment, monitoring, and governance emphasis CRISP-DM treats lightly |
| Project charter | A lightweight, whole-project companion to the Chapter 2 problem statement, naming stage owners and kill criteria |
| Pre-mortem | A structured exercise assuming the project already failed, to surface risks while still cheap to fix |
| Kill criteria | The explicit condition under which a project is cancelled rather than continued on sunk cost |
| Loop-back | A lifecycle transition to an earlier stage, triggered by what a later stage reveals: the norm, not the exception |
The Eight Stages, Mapped to This Book
| Stage | What happens | Where it lives |
|---|---|---|
| 1. Problem framing | Translate a business ask into a precise ML problem statement | Chapter 2 |
| 2. Data collection & understanding | Identify sources, pull data, understand structure/quality/limitations | Chapters 7-9 |
| 3. Data preparation | Clean, transform, engineer features into model-ready form | Chapters 7, 14 |
| 4. Modelling | Select algorithm family, train candidates, tune hyperparameters | Chapters 11-20 |
| 5. Evaluation | Measure against the stage-1 success metric: business metric, not just accuracy | Chapters 13, 16 |
| 6. Deployment | Package the model and serve predictions: batch or real-time | Book 2, Chapter 23 |
| 7. Monitoring | Watch for data drift, model drift, operational failures | Book 2, Chapter 25 |
| 8. Iteration | Feed evaluation/production learnings back into a revised problem statement | Back to stage 1 |
Framework Comparison
| Stage (this curriculum) | CRISP-DM | Microsoft TDSP |
|---|---|---|
| 1. Problem framing | Business Understanding | Business Understanding |
| 2. Data collection & understanding | Data Understanding | Data Acquisition & Understanding |
| 3. Data preparation | Data Preparation | Data Acquisition & Understanding (cleaning); Modeling (feature engineering) |
| 4. Modelling | Modeling | Modeling |
| 5. Evaluation | Evaluation | Modeling (evaluation sub-phase) |
| 6. Deployment | Deployment | Deployment |
| 7. Monitoring | (not explicit) | Deployment (ongoing) / Customer Acceptance |
| 8. Iteration | (implicit feedback loop) | Explicit: loops back to Business Understanding |
Key difference: CRISP-DM (1996, data-mining era) treats deployment lightly and has no explicit monitoring phase. TDSP (2016, cloud/MLOps era) treats deployment, monitoring, and governance as first-class stages. This book follows TDSP’s operational emphasis.
Stage Checklist: Inputs, Outputs, Exit Criteria
| Stage | Inputs | Outputs | Exit criteria |
|---|---|---|---|
| 1. Problem framing | Stakeholder ask | Filled problem statement | Label, features, horizon, success metric signed off |
| 2. Data understanding | Raw data sources | EDA report | Data quality, coverage, label reliability documented |
| 3. Data preparation | Raw/understood data | Model-ready dataset | No known leakage; split defined; pipeline reproducible |
| 4. Modelling | Model-ready dataset | Trained candidate model(s) | At least one model beats the baseline |
| 5. Evaluation | Trained model(s), held-out test set | Evaluation report | Model metric AND business metric both meet stage-1 thresholds |
| 6. Deployment | Approved model | Live serving endpoint or batch job | Model reachable in production; rollback plan exists |
| 7. Monitoring | Live system | Drift/performance dashboards + alerts | Named owner receives alerts; retraining trigger defined |
| 8. Iteration | Monitoring signals, retrospective | Revised problem statement | Feeds back into stage 1 |
Common Failure Modes by Stage
| Stage | Failure mode | Early warning sign |
|---|---|---|
| Problem framing | No kill criteria; ML used where a rule would do | “We’ll figure out success metrics later” |
| Data understanding | Label doesn’t reflect current business process | Label definition changed within the historical window |
| Data preparation | Leakage; feature not available at prediction time | Suspiciously high correlation with label () |
| Modelling | Optimising the wrong metric | Model metric improves but no business change follows |
| Evaluation | One-off evaluation, never repeated | No scheduled re-evaluation cadence |
| Deployment | Never leaves the notebook | No named deployment owner in the charter |
| Monitoring | No one owns production health | Alerts routed to no one, or an unread inbox |
| Iteration | “Done” treated as permanent | No retrospective run; no revised problem statement |
Decision Guide: Should This Be a Full Lifecycle Project?
Does a simple rule already meet the required performance?
|-- Yes → Ship the rule. Not a lifecycle project.
`-- No
|-- One-off analysis, no deployed system?
| |-- Yes → EDA/analytics task, not
| | a full lifecycle project.
| `-- No
| |-- Does historical labelled data exist
| | (or can it be constructed cheaply)?
| | |-- No → Data collection project first.
| | | Revisit once labels exist.
| | `-- Yes → Full lifecycle project.
| | Use the charter template.
Lifecycle Stage Owners (RACI Shorthand)
| Stage | Typically Responsible | Typically Accountable |
|---|---|---|
| Problem framing | Data scientist / SA | Business sponsor |
| Data understanding | Data scientist / data engineer | Data scientist |
| Data preparation | Data scientist / ML engineer | Data scientist |
| Modelling | Data scientist | Data scientist |
| Evaluation | Data scientist | Business sponsor (signs off on business metric) |
| Deployment | ML engineer / MLOps | Engineering lead |
| Monitoring | ML engineer / on-call | Named individual, not “the team” |
| Iteration | Whole team | Business sponsor |
AWS SageMaker Quick Reference
| Lifecycle stage | AWS service | Notes |
|---|---|---|
| Problem framing | (no AWS service: human process) | Document in the charter |
| Data collection & understanding | S3, Glue, Athena, SageMaker Data Wrangler | Chapters 7-9 tooling maps here |
| Data preparation | SageMaker Data Wrangler, SageMaker Feature Store | Chapter 14 |
| Modelling | SageMaker Training Jobs, SageMaker Experiments | Chapters 11-20 |
| Evaluation | SageMaker Clarify, SageMaker Model Monitor (baseline) | Chapter 16; Book 2, Chapter 21 (Clarify) |
| Deployment | SageMaker Endpoints (real-time), SageMaker Batch Transform | Book 2, Chapter 23 |
| Monitoring | SageMaker Model Monitor, CloudWatch | Book 2, Chapter 25 |
| Iteration / orchestration | SageMaker Pipelines, SageMaker Projects | Book 2, Chapter 24 |
Chapter 11 — Supervised Learning: Overview of Approaches
Key Terms
| Term | Definition |
|---|---|
| Inductive bias | An algorithm family’s built-in assumption about the shape of the relationship between inputs and outputs |
| Bias (statistical) | How wrong a model’s assumptions are, systematically, no matter how much data it gets |
| Variance (statistical) | How much a model’s predictions change when trained on a different sample from the same population |
| Curse of dimensionality | As dimensionality grows, all points become roughly equidistant, so “nearest neighbour” stops being meaningful |
| No Free Lunch theorem | Averaged over all possible problems, no algorithm outperforms any other. It does not rule out algorithms that reliably win on the specific kinds of problems seen in practice |
| Conditional independence | Naive Bayes’ assumption that features are independent of each other given the class label |
Algorithm Comparison Table
| Algorithm | Task | Interpretable | Non-linear | Needs scaling | Missing values | sklearn class |
|---|---|---|---|---|---|---|
| Logistic Regression | Classification | Yes | No (linear) | Yes | No | LogisticRegression |
| Linear Regression | Regression | Yes | No | No (yes once regularised) | No | LinearRegression |
| Ridge Regression | Regression | Yes | No | Yes | No | Ridge |
| Decision Tree | Both | Yes (if shallow) | Yes | No | Yes (sklearn 1.3+) | DecisionTreeClassifier / Regressor |
| K-Nearest Neighbours | Both | No | Yes | Yes (critical) | No | KNeighborsClassifier / Regressor |
| Naive Bayes | Classification | Partial | No | No | No | GaussianNB, MultinomialNB |
| Random Forest | Both | No | Yes | No | Yes (sklearn 1.4+) | RandomForestClassifier / Regressor |
| SVM | Both | No | Yes (with kernel) | Yes (critical) | No | SVC, SVR |
| XGBoost | Both | No | Yes | No | Yes (native) | xgboost.XGBClassifier |
The Universal sklearn API
model.fit(X_train, y_train) # train
# Class labels (classification) or values (regression)
model.predict(X_test)
# Class probabilities (classification only)
model.predict_proba(X_test)
model.score(X_test, y_test) # accuracy (clf) or R² (reg)
# Pipeline pattern (always use this)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([
('scaler', StandardScaler()),
('model', YourClassifier())
])
pipe.fit(X_train, y_train)
pipe.predict(X_test)Bias-Variance Quick Reference
| Symptom | Diagnosis | Fix |
|---|---|---|
| Train accuracy high, test accuracy low | High variance (overfitting) | More data, regularisation, simpler model, ensemble |
| Both train and test accuracy low | High bias (underfitting) | More complex model, better features, less regularisation |
| Large gap between CV folds | High variance | Reduce model complexity, more training data |
Algorithm Selection Decision Tree
What is your target?
|-- Discrete category → Classification
| |-- Need interpretability?
| | |-- Yes → LogisticRegression (linear) or
| | | DecisionTree (max_depth=4)
| | `-- No → RandomForest or XGBoost
| | (usually best on tabular)
| `-- Text data? → MultinomialNB or fine-tuned
| transformer
`-- Continuous number → Regression
|-- Linear relationship? → LinearRegression / Ridge
|-- Nonlinear? → RandomForestRegressor or XGBRegressor
`-- Time series? → Specialised forecasting models
Key Hyperparameters to Know
| Class | Key hyperparams | Notes |
|---|---|---|
LogisticRegression |
C (inverse regularisation, default 1.0), max_iter |
Larger C = less regularisation |
DecisionTreeClassifier |
max_depth, min_samples_leaf, min_samples_split |
Limit depth to prevent overfitting |
KNeighborsClassifier |
n_neighbors (default 5), metric, algorithm |
More neighbours = smoother boundary |
RandomForestClassifier |
n_estimators, max_depth, max_features |
Start with n_estimators=100 |
GaussianNB |
var_smoothing |
Rarely needs tuning |
Common Imports
from sklearn.linear_model import (
LogisticRegression,
LinearRegression,
Ridge,
)
from sklearn.tree import (
DecisionTreeClassifier,
DecisionTreeRegressor,
)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.ensemble import (
RandomForestClassifier,
RandomForestRegressor,
)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import (
train_test_split,
cross_val_score,
)
from sklearn.metrics import (accuracy_score, classification_report,
confusion_matrix, f1_score,
mean_squared_error, r2_score)AWS SageMaker Quick Reference
| Task | SageMaker approach | Notes |
|---|---|---|
| Linear/logistic regression at scale | SageMaker Linear Learner built-in algorithm | Distributed training for large tabular datasets |
| Tree ensembles at scale | SageMaker built-in XGBoost algorithm | Managed, spot-instance-friendly training jobs |
| k-NN at scale | SageMaker k-NN built-in algorithm | Approximate nearest-neighbour index for large datasets |
| Comparing algorithm families quickly | SageMaker Autopilot | Runs several candidate algorithms and reports a leaderboard, mirroring Procedure 11.2 |
Chapter 12 — Ensemble Methods
Ensemble Method Comparison
| Method | Strategy | Reduces | Best for | Library |
|---|---|---|---|---|
| Bagging | Bootstrap samples, average | Variance | High-variance base learners | BaggingClassifier |
| Random Forest | Bagging + random features | Variance | Tabular data, robust baseline | RandomForestClassifier/Regressor |
| sklearn Gradient Boosting | Sequential residual fitting | Bias + Variance | General tabular, slower | GradientBoostingClassifier/Regressor |
| XGBoost | GB + regularisation, pruning, sparsity | Bias + Variance | Most tabular tasks | xgboost.XGBClassifier/Regressor |
| LightGBM | Leaf-wise GB, GOSS, EFB | Bias + Variance | Large datasets, speed-critical | lightgbm.LGBMClassifier/Regressor |
| Stacking | Meta-learner on base predictions | Both | Squeezing last % accuracy | StackingClassifier/Regressor |
Key Hyperparameters
Random Forest: n_estimators (100, range 100-500), max_depth (None), max_features ('sqrt'), min_samples_leaf (1, range 1-20), n_jobs=-1.
XGBoost: n_estimators (100, range 100-1000), learning_rate/eta (0.3, range 0.01-0.3), max_depth (6, range 3-8), subsample (1.0, range 0.6-1.0), reg_alpha/reg_lambda (L1/L2).
LightGBM: n_estimators (100, range 100-2000), learning_rate (0.1, range 0.01-0.1), num_leaves (31, range 20-150, primary complexity control), min_child_samples (20, range 10-100).
Feature Importance Types
| Type | How computed | Strengths | Weaknesses |
|---|---|---|---|
| Impurity (Gini/variance) | Sum of impurity reduction at splits | Fast, no extra data | Biased towards high-cardinality features |
| Permutation | Drop in score when feature shuffled | No high-cardinality bias | Slower, needs held-out data; misleads on correlated features |
| SHAP | Shapley values | Per-prediction explanations | Requires shap library |
Common Imports
from sklearn.ensemble import (
RandomForestRegressor,
GradientBoostingRegressor,
StackingRegressor,
)
import xgboost as xgb
import lightgbm as lgb
from sklearn.inspection import permutation_importanceAWS SageMaker Quick Reference
| Task | SageMaker approach |
|---|---|
| Train XGBoost without a custom container | Built-in xgboost algorithm image |
| Distributed training on large tabular data | Built-in algorithm + multi-instance training job |
Chapter 13 — Cross-Validation & Model Selection
CV Strategy Selection Guide
| Situation | Strategy | sklearn class |
|---|---|---|
| Standard classification | Stratified k-fold (k=5) | StratifiedKFold |
| Standard regression | k-Fold (k=5) | KFold |
| Time-ordered data | Time-series split | TimeSeriesSplit |
| Tiny dataset (<50 samples) | Leave-One-Out | LeaveOneOut |
| Hyperparam tuning + evaluation | Nested CV | GridSearchCV inside cross_val_score |
| Groups (same subject in train+test) | Group k-fold | GroupKFold |
Common CV Pitfalls
| Pitfall | Consequence | Fix |
|---|---|---|
| Preprocessing fitted on whole data, not per-fold | Data leakage: overoptimistic scores | Always use Pipeline |
| Reporting test score after using it for selection | Overoptimistic: selection bias | Nested CV or a held-out final test set |
| Shuffled CV on time-series data | Leaks future into training | TimeSeriesSplit |
| Not stratifying for imbalanced classes | Unstable fold scores | StratifiedKFold |
Scoring Strings
| Scoring string | Metric | Task |
|---|---|---|
'accuracy' |
Fraction correct | Classification |
'f1_weighted' |
Weighted F1 | Multiclass |
'roc_auc' |
ROC-AUC | Binary classification |
'r2' |
R² | Regression |
'neg_root_mean_squared_error' |
−RMSE | Regression |
All regression scorers are negated, since sklearn maximises. Negate the result to get the real metric.
How Many Folds?
| Dataset size | Recommendation |
|---|---|
| < 50 samples | LeaveOneOut |
| 50-500 samples | k=10 |
| 500-10,000 samples | k=5 (default) |
| > 10,000 samples | k=3 or single split |
Pipeline + CV Pattern (Always Use This)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import (
cross_val_score,
StratifiedKFold,
)
pipe = Pipeline([
('scaler', StandardScaler()),
('model', YourModel()),
])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
pipe, X, y, cv=cv, scoring='accuracy', n_jobs=-1
)Common Imports
from sklearn.model_selection import (
train_test_split,
KFold,
StratifiedKFold,
TimeSeriesSplit,
LeaveOneOut,
cross_val_score,
cross_val_predict,
cross_validate,
GridSearchCV,
RandomizedSearchCV,
)Chapter 14 — Feature Engineering
Key Terms
| Term | Definition |
|---|---|
| Feature | A measurable property of an observation used as model input |
| Encoding | Converting a categorical variable into numbers |
| Cardinality | Number of unique values in a categorical column |
| Leakage | A feature (or its fitted statistics) using information the model shouldn’t have at prediction time |
| Cyclical encoding | sin/cos transform making a wrapping variable (hour, day of week) numerically continuous |
| Interaction feature | A feature built from combining two or more other features |
Encoding Strategy Comparison
| Encoding | Cardinality fit | Leakage risk | Model type |
|---|---|---|---|
| One-hot | Low (< 20) | None | Any, especially linear/distance-based |
| Ordinal | Low, ordered | None | Any |
Integer codes (OrdinalEncoder) |
Any | None | Tree-based only |
| Frequency | High | None | Any |
| Target / mean (CV) | High | High: must use cross-fitting | Any |
Scaler Comparison
| Scaler | Formula | Sensitive to outliers? | Use when |
|---|---|---|---|
StandardScaler |
Yes | Default, roughly normal data | |
MinMaxScaler |
/ | Very | Bounded input needed |
RobustScaler |
No | Data has significant outliers | |
| None | — | — | Tree-based models |
Decision Guide
Is the column categorical?
|-- Yes
| |-- Cardinality < 20?
| | |-- Has natural order? → OrdinalEncoder
| | `-- No order → OneHotEncoder
| `-- Cardinality >= 20?
| |-- Tree-based model? → OrdinalEncoder
| `-- Linear/other model?
| → Frequency or CV target encoding
`-- No (numeric)
|-- Datetime?
| → extract hour/dow/month + cyclical sin/cos
|-- Right-skewed? → log1p transform
`-- Feeding linear/distance model?
→ StandardScaler / RobustScaler
Common Imports
from sklearn.preprocessing import (
OneHotEncoder,
OrdinalEncoder,
StandardScaler,
PolynomialFeatures,
KBinsDiscretizer,
)
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputerAWS SageMaker Quick Reference
| Task | SageMaker approach |
|---|---|
| No-code feature transforms | SageMaker Data Wrangler flow |
| Shared feature store (train + serve) | SageMaker Feature Store |
| Embed preprocessing in inference | sklearn.Pipeline saved via joblib, deployed as one model |
Chapter 15 — Feature Selection & Dimensionality Reduction
Key Terms
| Term | Definition |
|---|---|
| Feature selection | Choosing a subset of existing features; the rest are discarded, survivors keep their original meaning |
| Feature extraction | Constructing a smaller number of new features as combinations of the originals (e.g. PCA components) |
| Curse of dimensionality | The set of problems (sparse data, meaningless distances, overfitting) that worsen as feature count grows |
| Explained variance ratio | Fraction of total dataset variance captured by a given principal component |
| Loading | The weight of an original feature in a principal component’s linear combination |
| Perplexity (t-SNE) | Roughly, the effective number of nearest neighbours considered per point; controls local vs. global balance |
| Embedding | The low-dimensional representation produced by PCA/t-SNE/UMAP |
Selection Method Comparison
| Method | Family | Model-aware? | Handles non-linear relationships? | Relative cost |
|---|---|---|---|---|
| Variance threshold | Filter | No | N/A (variance only) | Very low |
| Correlation with target | Filter | No | No (linear only) | Very low |
| Mutual information | Filter | No | Yes | Low-medium |
| RFE / RFECV | Wrapper | Yes | Depends on base estimator | High (retrains repeatedly) |
| L1 / Lasso coefficients | Embedded | Yes | No (linear model) | Low (one fit) |
| Tree feature importance | Embedded | Yes | Yes | Low-medium (one fit) |
PCA vs. t-SNE vs. UMAP
| Property | PCA | t-SNE | UMAP |
|---|---|---|---|
| Linear or non-linear | Linear | Non-linear | Non-linear |
| Preserves | Global variance | Local neighbourhood structure | Local + more global structure than t-SNE |
| Deterministic | Yes | No (stochastic) | No (stochastic, but more stable) |
.transform() on new data |
Yes | No (classic implementation) | Yes (with caveats) |
| Interpretable components | Somewhat (loadings) | No | No |
| Typical use | Preprocessing for modelling, or visualisation | Visualisation only | Visualisation, sometimes clustering preprocessing |
| Relative speed on large data | Fast | Slow | Faster than t-SNE |
Decision Guide
Need the output to feed a downstream model (not a plot)?
|-- Yes
| |-- Want to keep original feature meaning?
| | → Feature selection (filter/wrapper/embedded)
| `-- OK losing interpretability for compactness?
| → PCA
`-- No — just exploring / visualising
|-- Fast look, large dataset (100k+ rows)? → UMAP
|-- Best local cluster separation, small-medium
| dataset? → t-SNE
`-- Fast, deterministic, reproducible 2D view? → PCA
API / Function Summary
# Features with variance <= threshold are removed
VarianceThreshold(threshold=0.0)
SelectKBest(
# or f_classif, chi2, mutual_info_regression, f_regression
score_func=mutual_info_classif,
k=10,
)
mutual_info_classif(
X, y, discrete_features='auto', random_state=42
)
RFE(estimator, n_features_to_select=10, step=1)
# .support_, .ranking_, .n_features_
RFECV(estimator, step=1, cv=5, scoring='accuracy')
SelectFromModel(estimator, prefit=False, threshold='median')
PCA(
# int, float in (0,1) (variance target), or None (all)
n_components=None,
random_state=42,
svd_solver='auto',
)
# .explained_variance_ratio_, .components_ (loadings),
# .n_components_
# Like PCA but for sparse matrices, no centering
TruncatedSVD(n_components=10, random_state=42)
# PCA for data with non-linear structure; the kernel trick
# from Chapter 11's SVM section, applied to PCA
KernelPCA(n_components=2, kernel='rbf', gamma=None, random_state=42)
# No simple inverse back to the original feature space
TSNE(
n_components=2,
perplexity=30,
learning_rate='auto',
random_state=42,
max_iter=1000,
)
# No .transform() for new points in the classic implementation
UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
# supports .transform() on new data, unlike classic t-SNECommon Imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import (
VarianceThreshold,
SelectKBest,
mutual_info_classif,
mutual_info_regression,
RFE,
RFECV,
SelectFromModel,
)
from sklearn.decomposition import PCA, TruncatedSVD, KernelPCA
from sklearn.manifold import TSNE
from sklearn.ensemble import RandomForestClassifier
# pip install umap-learn
import umapAWS SageMaker Quick Reference
| Task | SageMaker approach | Documentation |
|---|---|---|
| Distributed PCA on very large datasets | Built-in PCA algorithm |
sagemaker.PCA estimator |
| No-code feature importance / correlation analysis | SageMaker Data Wrangler analyses | sagemaker.wrangler.processing.DataWranglerProcessor |
| Interactive exploratory dimensionality reduction | SageMaker Studio notebook with sklearn/UMAP | Standard notebook instance / Studio kernel |
| Feature selection as a repeatable pipeline step | SageMaker Processing job (SKLearnProcessor) wrapping the fitted selector | sagemaker.sklearn.processing.SKLearnProcessor |
Chapter 16 — Model Evaluation
Classification Metrics — Formulas
| Metric | Formula | Higher = better? |
|---|---|---|
| Accuracy | Yes | |
| Precision | Yes | |
| Recall (Sensitivity) | Yes | |
| F1 Score | Yes | |
| ROC-AUC | Area under ROC curve | Yes (0.5 = random) |
| PR-AUC / Avg Precision | Area under PR curve | Yes |
| Log Loss | No (lower better) |
Regression Metrics — Formulas
| Metric | Formula | Sensitive to outliers? |
|---|---|---|
| MSE | Yes | |
| RMSE | Yes | |
| MAE | Less | |
| MAPE | Less | |
| Moderate |
RMSE/MAE ratio > 2 indicates outliers are significantly influencing RMSE.
Metric Selection Guide
| Business context | Recommended metric |
|---|---|
| Balanced classes, symmetric errors | Accuracy or F1 |
| Imbalanced classes | F1, PR-AUC |
| FP expensive (spam filter) | Precision |
| FN expensive (cancer, fraud) | Recall |
| Compare models before setting threshold | ROC-AUC |
| Probabilities matter (pricing, risk) | Log Loss + calibration |
| Rare event detection | PR-AUC (ROC-AUC is misleading) |
Calibration Interpretation
| Curve shape | Meaning | Fix |
|---|---|---|
| On the diagonal | Well-calibrated | Nothing needed |
| S-curve (probabilities avoid 0 and 1) | Under-confident at the extremes; common in Random Forests | Isotonic regression or Platt (sigmoid) scaling |
| Below diagonal | Over-estimates: observed rate is lower than predicted | Isotonic regression or Platt (sigmoid) scaling |
sklearn API Quick Reference
from sklearn.metrics import (
accuracy_score,
precision_score,
recall_score,
f1_score,
roc_auc_score,
average_precision_score,
log_loss,
confusion_matrix,
classification_report,
roc_curve,
precision_recall_curve,
mean_squared_error,
mean_absolute_error,
r2_score,
)
from sklearn.calibration import (
calibration_curve,
CalibratedClassifierCV,
)AWS SageMaker Quick Reference
| Task | Approach |
|---|---|
| Log metrics during training | sagemaker.Session().log_metric(name, value, step) |
| Automatic model quality monitoring | SageMaker Model Monitor: ModelQualityMonitor |
Chapter 17 — Hyperparameter Optimisation
Search Strategy Comparison
| Strategy | Pros | Cons | Use when |
|---|---|---|---|
| Manual (one-at-a-time) | Simple, interpretable | Misses interactions | Debugging only |
| Grid Search | Exhaustive over grid | Exponential in #params | 3 params, fast model |
| Random Search | Wide coverage, parallelisable | No guidance from past trials | Most situations |
| Bayesian (Optuna) | Sample-efficient, prunes bad trials | Trials run in sequence | Expensive models |
Rule of thumb: random search with 50-100 trials beats grid search on most problems, especially once a grid is restricted to just a couple of values per dimension. This chapter’s Worked Example measures the gap directly and shows why: a coarse grid can only sample the corners of the search space, never the interior.
Key Hyperparameters by Algorithm
XGBoost/LightGBM: n_estimators (100-1000), max_depth (3-8), learning_rate (0.01-0.3, log scale), subsample (0.6-1.0), reg_alpha/reg_lambda.
Random Forest: n_estimators (100-1000), max_depth (5-30 or None), max_features ('sqrt', 'log2'), min_samples_leaf (1-10).
Logistic Regression: C (0.001-100, log scale), penalty ('l1', 'l2', 'elasticnet').
Optuna Quick-Start
import optuna
optuna.logging.set_verbosity(optuna.logging.WARNING)
def objective(trial):
params = {
'n_estimators': trial.suggest_int(
'n_estimators', 50, 500
),
'max_depth': trial.suggest_int('max_depth', 2, 10),
'learning_rate': trial.suggest_float(
'learning_rate', 0.01, 0.3, log=True
),
}
model = MODEL_CLASS(**params, random_state=42)
scores = cross_val_score(
model, X_train, y_train, cv=5, scoring='roc_auc'
)
return scores.mean()
study = optuna.create_study(
direction='maximize',
sampler=optuna.samplers.TPESampler(seed=42),
)
study.optimize(objective, n_trials=100)Optuna suggest_* API
trial.suggest_int('name', low, high)
trial.suggest_float('name', low, high)
trial.suggest_float('name', low, high, log=True)
trial.suggest_categorical('name', ['a', 'b', 'c'])Validation Set Hygiene
Training data → hyperparameter search (CV folds)
→ pick best config
Test data → evaluate best model ONCE, at the very end
Never tune on the test set. After tuning, the CV score is optimistic. The test score is the honest estimate.
AWS SageMaker HPO Quick Reference
| Feature | SageMaker equivalent |
|---|---|
| Bayesian search | strategy='Bayesian' in HyperparameterTuner |
| Random search | strategy='Random' |
| Parallel trials | max_parallel_jobs=N |
Chapter 18 — Regularisation
Regularisation Method Comparison
| Method | Penalty | Zeroes coefficients? | Best for |
|---|---|---|---|
| Ridge (L2) | No: shrinks all | Most features informative, correlated features | |
| Lasso (L1) | Yes: sparse solution | Many irrelevant features, automatic selection | |
| Elastic Net | Yes | Correlated features + sparsity | |
| Dropout | Random zero-out during training | N/A | Neural networks |
| Early stopping | Stop when val loss stops improving (with patience) | N/A | Neural nets, gradient boosting |
Bias-Variance Trade-off
| Low Complexity | High Complexity | |
|---|---|---|
| Bias | High | Low |
| Variance | Low | High |
| Test Error | High (“underfitting”) | High (“overfitting”) |
Learning Curve Diagnosis
| Pattern | Diagnosis | Fix |
|---|---|---|
| Both curves converge to a similarly high error | High bias | More flexible model, better features |
| Train error stays low, validation error stays higher, gap persists | High variance | More data, more regularisation |
Key Parameters
| Model | Regularisation param | Direction |
|---|---|---|
| Ridge / Lasso | alpha |
Larger = more regularisation |
| ElasticNet | alpha, l1_ratio |
l1_ratio: 0=Ridge, 1=Lasso |
| LogisticRegression | C |
Smaller C = more regularisation () |
| XGBoost/LightGBM | reg_alpha (L1), reg_lambda (L2) |
Larger = more |
Regularisation in Other Model Types
| Model type | L2 equivalent | L1 equivalent |
|---|---|---|
| XGBoost/LightGBM | reg_lambda |
reg_alpha |
| Neural networks | weight decay in optimiser | None (dropout is the main regulariser) |
| Decision trees | max_depth, min_samples_leaf (structural) |
— |
sklearn API
from sklearn.linear_model import (
Ridge, RidgeCV, Lasso, LassoCV, ElasticNet, ElasticNetCV,
LogisticRegression, LogisticRegressionCV,
)Dropout Quick Reference
import torch.nn as nn
# Zero out 50% of activations during training
nn.Dropout(p=0.5)
# model.train() -> dropout active
# model.eval() -> dropout disabledAWS SageMaker Equivalent
from sagemaker.linear_learner import LinearLearner
ll = LinearLearner(
predictor_type='binary_classifier', l1=0.01, wd=0.001
)Chapter 19 — Unsupervised Learning
Key Terms
| Term | Definition |
|---|---|
| Clustering | Grouping similar points without ground-truth labels |
| Anomaly detection | Identifying points that don’t fit the pattern of the rest of the data |
| Centroid | Mean position of all points assigned to a cluster (K-Means) |
| Inertia | Sum of squared distances from each point to its assigned centroid |
| Core point | A point with at least min_samples neighbours within eps (DBSCAN) |
| Contamination | Expected fraction of anomalies in a dataset |
| BIC / AIC | Bayesian/Akaike Information Criterion: single scores for choosing GMM component count; BIC penalises extra components more as data grows, AIC keeps the penalty fixed |
Algorithm Comparison
| Approach | Best for | Weakness |
|---|---|---|
| K-Means | Fast, large datasets, roughly spherical clusters | Must specify K; assumes equal-size, spherical clusters |
| Gaussian Mixture Model | Elliptical clusters, soft/probabilistic membership | Slower (full covariance per component); can still converge to a poor local optimum |
| DBSCAN | Arbitrary-shaped clusters, automatic outlier detection | Sensitive to eps/min_samples; struggles with varying density |
| Agglomerative | Small datasets, exploratory analysis, dendrograms | At least time and memory: impractical above ~10k rows |
| Isolation Forest | High-dimensional anomaly detection, scales well | Random results across runs; needs contamination estimate |
Clustering Evaluation Metrics
| Metric | Range | Higher better? | Use when |
|---|---|---|---|
| Silhouette score | -1 to 1 | Yes | General-purpose cluster quality |
| Davies-Bouldin index | 0 to ∞ | No | Comparing compactness/separation across K |
| Calinski-Harabasz index | 0 to ∞ | Yes | Fast, good for large datasets |
| Adjusted Rand Index | -0.5 to 1 | Yes | Only with ground-truth labels available |
Decision Guide
Roughly know cluster count, likely spherical? → K-Means
Need soft membership or elliptical clusters? → Gaussian Mixture Model
Unknown K, or irregular shapes / noise? → DBSCAN
Small dataset (<10k), want to explore visually?
→ Agglomerative + dendrogram
High-dim, large anomaly-detection dataset?
→ IsolationForest
Density varies a lot across feature space?
→ LocalOutlierFactor
Small, clean dataset, tight non-linear boundary?
→ OneClassSVM
Common Imports
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.mixture import GaussianMixture
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor, NearestNeighbors
from sklearn.metrics import (
silhouette_score,
davies_bouldin_score,
adjusted_rand_score,
)AWS SageMaker Quick Reference
| Task | SageMaker approach |
|---|---|
| Managed K-Means at scale | sagemaker.KMeans built-in algorithm |
| Managed anomaly detection | sagemaker.RandomCutForest built-in algorithm |
| Real-time anomaly scoring on streams | Managed Service for Apache Flink calling a deployed model (Kinesis Data Analytics for SQL, with RANDOM_CUT_FOREST, retired in January 2026) |
Chapter 20 — Time Series Analysis
Key Terms
| Term | Definition |
|---|---|
| Trend | The long-run direction of the series |
| Seasonality | A pattern that repeats at a fixed, known period |
| Stationarity | Statistical properties (mean, variance, autocorrelation) constant over time |
| Differencing | Subtracting a lagged value to remove trend or seasonality |
| ACF / PACF | Autocorrelation / partial autocorrelation functions |
| Unit root | Property of a non-stationary series; the ADF test’s null hypothesis |
ACF / PACF → Model Order Cheat Sheet
| Pattern | Suggests |
|---|---|
| ACF cuts off sharply after lag q, PACF decays gradually | MA(q) |
| PACF cuts off sharply after lag p, ACF decays gradually | AR(p) |
| Both decay gradually | Mixed ARMA(p,q): use auto_arima |
| ACF decays very slowly | Non-stationary: difference first |
| Spike at lag m and multiples of m | Seasonal component at period m |
Forecasting Method Comparison
| Approach | Best for | Weakness |
|---|---|---|
| Naive / seasonal naive | Baseline: always compute first | No real “modelling” |
| Exponential smoothing | Simple, robust, minimal tuning | Limited to trend + one seasonal pattern |
| ARIMA / SARIMA | Single series, built-in prediction intervals | Requires stationarity, manual/auto order selection |
| Lag-feature gradient boosting | Many related series, external regressors | No native uncertainty intervals |
Forecast Evaluation Metrics
| Metric | Formula | Use when |
|---|---|---|
| MAE | Default; robust to outliers | |
| RMSE | Penalise large errors more | |
| MAPE | Cross-scale comparison; avoid near zero | |
| sMAPE | Bounded 0-200% variant of MAPE | More stable near zero |
API Quick Reference
from statsmodels.tsa.seasonal import seasonal_decompose
from statsmodels.tsa.stattools import adfuller
from statsmodels.tsa.statespace.sarimax import SARIMAX
from sklearn.model_selection import TimeSeriesSplit
import pmdarima as pm
decomp = seasonal_decompose(x, model='additive', period=7)
# Reject non-stationarity if p < 0.05
adf_stat, p_value, *_ = adfuller(x, autolag='AIC')
model = SARIMAX(endog, order=(p,d,q), seasonal_order=(P,D,Q,m))
fitted = model.fit(disp=False)
forecast = fitted.get_forecast(steps=28)Decision Guide
Stationary (ADF p < 0.05)?
|-- No → difference it, re-test
`-- Yes
|-- Single series, need uncertainty intervals?
| → ARIMA/SARIMA
|-- Many related series / external features?
| → Lag-feature gradient boosting
|-- Missing data / holiday effects? → Prophet
`-- Very large dataset, many series?
→ Deep learning (LSTM / TFT)
AWS SageMaker Quick Reference
| Task | SageMaker approach |
|---|---|
| AutoML forecasting across many series | SageMaker Canvas time-series forecasting (Amazon Forecast, create_auto_predictor, for existing accounts only) |
| Custom deep learning forecaster | SageMaker built-in DeepAR algorithm |
| No-code forecasting | SageMaker Canvas |