Chapter 12
Ensemble Methods
A tree standing alone is easily blown down in a storm, where a forest continues to stand. A single decision tree is weak in the same way. Train it on slightly different data, and the result is a completely different tree. Train it deep enough to capture real complexity, and it overfits. Keep it shallow enough to generalise, and it underfits. Yet tree-based algorithms, combined into ensembles, produce the most accurate models for tabular data that exist. This chapter is about why combining many mediocre models consistently beats any single good one, and how that principle plays out differently in bagging, boosting, and stacking.
12.1 Motivation
The underlying intuition is old: wisdom of crowds. In 1906, statistician Francis Galton looked at about 800 guesses of an ox’s weight from a county fair contest. The median guess landed within 1% of the true weight, closer than most of the individual guesses. Individual errors cancelled out in the aggregate. Ensemble methods apply the same principle to machine learning models. But it only works if the individual models make different errors. If every model in the ensemble fails on the same examples, averaging doesn’t help. The diversity of errors is what produces the advantage. There are two main strategies for creating that diversity: bagging and boosting.
12.2 Exposition
12.2.1 Bagging — Reducing Variance
Bagging stands for Bootstrap AGGregating. Sample examples from the training set with replacement. Each example goes back into the pool before the next draw, so some appear multiple times and some not at all. Train a model on this bootstrap sample, repeat times, then average the regression outputs (or majority-vote the classification outputs). Each model sees a slightly different dataset and makes slightly different errors. Averaging cancels the errors and amplifies the signal. If models each have variance and their errors are uncorrelated, the average has variance : more models, lower variance, with no increase in bias.
12.2.2 Random Forests — Bagging Plus Feature Randomness
Random Forests add a second source of diversity on top of bagging: at each split in each tree, only a random subset of the features is considered: roughly of for classification. (scikit-learn’s RandomForestRegressor defaults to all features, which makes it plain bagging unless max_features is set.) This sounds like a bad idea. Why limit the information available to each split? Because without feature randomness, every tree in a bagged ensemble tends to pick the same dominant feature at the root, producing correlated trees whose errors don’t cancel. Feature randomness decorrelates the trees. Decorrelated trees produce more diverse errors. More diverse errors cancel better when averaged. That’s why Random Forests are usually more accurate than plain bagging. Random Forests are especially strong on tabular data with mixed feature types, when feature importance scores are wanted, and when there’s little appetite for hyperparameter tuning.
12.2.3 Boosting — Reducing Bias
Boosting trains models sequentially rather than in parallel, with each new model focused on the previous ensemble’s mistakes. The original recipe, AdaBoost: train a weak learner (typically a shallow “stump”), increase the weight of misclassified examples, train the next learner on the reweighted data so it focuses on the hard cases, repeat, and combine as a weighted sum. Boosting mainly reduces bias, systematic error. Gradient boosting’s shrinkage and subsampling also keep its variance in check. It’s especially effective when a model is wrong in a correctable way.
12.2.4 Gradient Boosting — Boosting as Gradient Descent
Modern gradient boosting (Friedman, 2001) reframes boosting as gradient descent in function space. Ordinary gradient descent (Chapter 4) nudges a set of numbers, the weights, in the direction that most reduces the loss. Gradient boosting nudges a whole function instead: each new tree is the “step,” added to the ensemble because it points the current prediction function in the direction that reduces error the most. Instead of reweighting examples, each new tree is trained to predict the residual errors of the current ensemble (for squared error; for other losses, the negative gradient of the loss): fit tree to predict , compute residuals, fit tree to predict residuals, and continue. Each tree corrects what the ensemble currently gets wrong, and the training error falls with each round. The learning rate (shrinkage) controls how aggressively each tree corrects: small learning rates need more trees but generalise better.
12.2.5 XGBoost and LightGBM
XGBoost (2016, Chen & Guestrin) is an engineered gradient boosting implementation adding built-in L1/L2 regularisation on leaf weights, pruning (grows trees then prunes back, rather than greedy early stopping), native handling of missing values, row/column subsampling, and parallelised split-finding. Its paper reports that 17 of the 29 winning Kaggle solutions published in 2015 used it.
LightGBM (2017, Microsoft) improves training speed in three ways. It grows trees leaf-wise instead of level-wise, finding whichever split reduces loss most, whatever the resulting depth. It also uses Gradient-based One-Side Sampling (keeping high-gradient examples, sampling low-gradient ones), and Exclusive Feature Bundling (merging mutually exclusive sparse features). Its original paper reports training up to 20x faster than earlier gradient boosting implementations on large datasets.
The key insight for a Solutions Architect: on tabular data (the vast majority of enterprise ML), gradient boosting (XGBoost or LightGBM) tends to beat neural networks on accuracy. It also trains faster, is easier to interpret via SHAP, and costs less to run. Benchmark against LightGBM before reaching for a deep learning solution on structured data.
SHAP (SHapley Additive exPlanations) breaks a single prediction down into each feature’s contribution, in a way that’s mathematically guaranteed to add up to the gap between that prediction and the model’s average prediction, for any model type. For a rejected loan application, SHAP can say exactly how many percentage points “low credit score” or “short employment history” each contributed, not just that the model used those features somewhere. What isn’t guaranteed everywhere is exactness of the individual numbers. Computing the true Shapley value means checking every possible subset of features, exponential in general. Tree-based ensembles are the exception: TreeExplainer computes exact values in polynomial time, by exploiting the tree structure directly. That’s a real part of why they’re easier to interpret in practice than a neural network of comparable accuracy, whose SHAP values come from a slower approximation instead. Book 2, Chapter 21 covers SHAP properly, including how to compute and read it.
12.2.6 Stacking — Learning How to Combine
Stacking (Wolpert, 1992) trains a meta-learner to decide how to weight the base models’ predictions, rather than averaging or majority-voting them. Train base models using cross-validation. Use their out-of-fold predictions as features for a meta-learner (typically logistic or ridge regression). Train the meta-learner on those features against the true labels. For new data, generate base predictions and pass them to the meta-learner. Stacking is most valuable in competition settings. The complexity it adds is often not worth it in production, and, as this chapter’s Worked Example shows directly, it is not automatically better than the best individual model.
12.3 Trade-offs and Limits
| Method | Reduces | Risk | Best for |
|---|---|---|---|
| Bagging | Variance | None inherent | High-variance base learners (deep trees) |
| Random Forest | Variance | Slow on very wide data | Robust tabular baseline |
| Gradient Boosting | Bias + Variance | Overfitting if trees too deep / learning rate too high | Most tabular tasks |
| Stacking | Both, in principle | Complexity, leakage if done wrong, no guarantee of improvement | Squeezing the last per cent in competitions |
Random Forests and boosted trees both work well on tabular data. But they are not interchangeable defaults. Random Forests are more forgiving of minimal tuning. Gradient boosting variants need more care, learning rate, tree depth, number of estimators, to reach their best accuracy. That same sensitivity is what lets them beat Random Forests once tuned.
12.4 Procedures
Procedure 12.1 (Tune n_estimators and max_depth). When to use: after choosing an ensemble method, to improve its accuracy.
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
for n in [50, 100, 200, 300]:
scores = cross_val_score(
RandomForestRegressor(
n_estimators=n, random_state=42, n_jobs=-1
),
X_train,
y_train,
cv=5,
scoring='neg_root_mean_squared_error',
)
rmse = -scores.mean()
print(f'n={n:>3} RMSE={rmse:.0f} ± {scores.std():.0f}')Guidance: for Random Forest, n_estimators=100 is a safe default, and going beyond 300 rarely helps. For XGBoost/LightGBM, use more trees (500-1000) with a smaller learning rate (0.05). max_depth=4 is a reasonable starting point for gradient boosting. Random Forest typically uses unlimited depth (None).
Procedure 12.2 (Plot Feature Importance). When to use: to explain which features drive a model’s predictions.
# Impurity-based, all sklearn ensembles
importances = model.feature_importances_
from sklearn.inspection import permutation_importance
result = permutation_importance(
model, X_test, y_test, n_repeats=10, random_state=42
)Common mistake: trusting impurity-based .feature_importances_ alone. It’s biased towards high-cardinality features. Permutation importance (recomputing performance with a feature shuffled) is slower but doesn’t favour high-cardinality features, and should be the tiebreaker whenever the two disagree. It can still mislead when features are strongly correlated with each other.
Procedure 12.3 (Choose Between Random Forest and XGBoost/LightGBM). When to use: starting a new tabular modelling task.
# No time to tune?
# -> Random Forest (n_estimators=100, defaults work well)
# Training speed critical on a large dataset? -> LightGBM
# Missing values present?
# -> XGBoost or LightGBM (handle natively)
# Need native categorical support?
# -> LightGBM (categorical_feature=...) or CatBoostProcedure 12.4 (Build a Simple Stacking Ensemble). When to use: squeezing the last 0.5-1% accuracy out of a set of already-good models, not as a default choice.
from sklearn.model_selection import cross_val_predict, KFold
from sklearn.linear_model import Ridge
kf = KFold(n_splits=5, shuffle=True, random_state=42)
base_models = {
'rf': RandomForestRegressor(
n_estimators=100, random_state=42, n_jobs=-1
),
'lgb': lgb.LGBMRegressor(
n_estimators=100, learning_rate=0.1, random_state=42
),
}
oof, test_preds = {}, {}
for name, model in base_models.items():
oof[name] = cross_val_predict(model, X_train, y_train, cv=kf)
model.fit(X_train, y_train)
test_preds[name] = model.predict(X_test)
meta = Ridge(alpha=1.0).fit(
np.column_stack(list(oof.values())), y_train
)
y_pred = meta.predict(
np.column_stack(list(test_preds.values()))
)Common mistake: training the meta-learner on the base models’ predictions from the same data the base models were trained on. That causes leakage: always use out-of-fold predictions (cross_val_predict), never .predict() on the training set.
12.5 Worked Example
Racing four ensemble methods on a synthetic house-price dataset gives a direct, measured comparison rather than a textbook ranking. The data has 2,000 houses and 14 features: square footage, room and bathroom counts, lot size, age, garage, pool, floors, distance to school, local crime rate, neighbourhood, condition, style, and heating type. The split is 80/20 train/test.
models = {
'Random Forest': RandomForestRegressor(
n_estimators=200, random_state=42, n_jobs=-1
),
'Gradient Boosting': GradientBoostingRegressor(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
random_state=42,
),
'XGBoost': xgb.XGBRegressor(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
random_state=42,
),
'LightGBM': lgb.LGBMRegressor(
n_estimators=200,
learning_rate=0.1,
max_depth=4,
random_state=42,
),
}
for name, model in models.items():
model.fit(X_train, y_train)
# time and score each modelMeasured results:
| Model | RMSE | R² | Train time |
|---|---|---|---|
| Random Forest | $45,502 | 0.8451 | 0.15s |
| Gradient Boosting | $28,595 | 0.9388 | 0.47s |
| XGBoost | $28,873 | 0.9376 | 0.68s |
| LightGBM | $28,323 | 0.9400 | 0.06s |
LightGBM is both the fastest to train (0.06s, roughly 2.5x faster than Random Forest and 8-11x faster than the other two boosted methods) and the most accurate here, edging out XGBoost’s $28,873 and Gradient Boosting’s $28,595. The “start with LightGBM” guidance earns its keep on both counts in this run, not just speed. Random Forest, with no boosting at all, is roughly $17,000 RMSE behind the boosted methods, a direct illustration of boosting’s bias-reduction advantage over bagging on a task like this.
Feature importance (Random Forest) confirms sqft dominates as designed (0.551), with neighborhood a clear second (0.160). Below that the picture is less tidy than a designed dataset might suggest. n_rooms (0.065) and school_dist (0.060) both edge out condition (0.036), and lot_size (0.033) isn’t negligible either, though the price formula weights it far more lightly than square footage. That’s a useful reminder that a feature’s importance to the model and its weight in the underlying formula aren’t the same thing. Which features a tree-based model finds useful depends on how much separable signal they carry in this specific sample, not just how heavily they’re weighted in the process that generated the data.
Sweeping n_estimators from 5 to 500 shows the two algorithms converging differently. LightGBM’s RMSE drops sharply from $87,831 at 5 trees to $28,323 at 200 trees before flattening. Random Forest is already within a few thousand dollars of its final RMSE at 25 trees ($47,511 vs. $45,502 at 200). Bagging’s variance reduction saturates fast, while boosting needs enough rounds to substantially reduce bias.
Finally, stacking a Random Forest and a LightGBM model with a Ridge meta-learner produced RMSE $29,535 (R² 0.9347). That’s worse than solo LightGBM ($28,323), despite LightGBM being one of the two base models feeding the stack. The meta-learner’s own weights explain why: Ridge assigns LightGBM a weight of 1.209 and Random Forest a weight of -0.184, meaning it learned to partly subtract Random Forest’s much weaker predictions rather than blend them in. Even with that correction, including a clearly weaker model cost more than it gave back. This is a real, measured result, not a contrived counterexample. Stacking a strong model together with a much weaker one has no guarantee of beating the strong model alone. Procedure 12.4’s “not a default choice” framing is exactly this finding in practice, not a hedge.
# XGBoost is a SageMaker built-in algorithm - no custom
# training script needed
import sagemaker
from sagemaker.estimator import Estimator
xgb_estimator = Estimator(
image_uri=sagemaker.image_uris.retrieve(
"xgboost", region, version="1.7-1"
),
instance_type="ml.m5.xlarge",
instance_count=1,
hyperparameters={
"num_round": 200, "max_depth": 4, "eta": 0.1
},
)
xgb_estimator.fit({
"train": train_s3_uri, "validation": val_s3_uri
})When to prefer the built-in SageMaker XGBoost algorithm: when the team wants managed infrastructure, built-in distributed training for large datasets, and hyperparameter tuning jobs, without maintaining a custom training container. The trade-off is less flexibility than a hand-written xgb.XGBRegressor script for unusual preprocessing or custom objectives.
12.6 History and Further Reading
Murphy’s Introduction, sections 18.1-18.4, gives the formal treatment of bagging, random forests, and boosting (Murphy 2022). Thakur, Ch. 6, covers practical gradient boosting with code (Thakur 2020). Burkov, Ch. 7, gives a crisp ensemble overview (Burkov 2019). Géron’s Hands-On Machine Learning, Ch. 7, covers bagging, random forests, and boosting (AdaBoost and gradient boosting) together with the Scikit-Learn code for each (Géron 2019). The XGBoost paper is worth reading for the engineering behind its speed (Chen and Guestrin 2016).
12.7 How This Connects
This chapter builds on Chapter 11’s decision trees as base learners. It looks ahead to Chapter 13’s cross-validation, which is critical for stacking done correctly. It enables Chapter 14’s feature importance work and Chapter 16’s evaluation of ensemble models. It’s often confused with neural networks as competing options for tabular data. Gradient boosting usually wins on tabular data. Neural networks win on images, text, and audio, a distinction Book 2 returns to from Chapter 27 onward.
12.8 Exercises
[W] Explain in your own words why bagging reduces variance without increasing bias, using the relationship from this chapter’s Exposition.
[W] State which of bagging or boosting each of the following targets: (a) a model that is systematically wrong on a specific subgroup of examples; (b) a model whose predictions change wildly with small changes to the training data.
[A] Using Listing 12.5’s setup, explain why Random Forest’s RMSE barely changes between
n_estimators=200andn_estimators=500($45,502 vs. $45,514) while LightGBM’s RMSE at the same two settings keeps improving slightly, from $28,323 to $28,159. What does this suggest about each method’s sensitivity to over-provisioning trees?[A] Using Listing 12.4 as a base, add XGBoost as a third base model to the stack. State, from this chapter’s own measured numbers, whether you’d expect the resulting stack to beat solo XGBoost’s $28,873 RMSE.
[E] A customer’s data science team reports “we stacked five models and it’s more accurate than any of them individually.” Using this chapter’s Worked Example, write the two or three sentences you’d say before accepting that claim.
[E] A colleague argues Random Forest should always be tried before XGBoost “because it’s simpler.” Using Procedure 12.3 and this chapter’s measured accuracy gap (Random Forest’s RMSE roughly 58% higher than XGBoost’s), write the case for when that advice is right and when it isn’t.
[R] Read the XGBoost paper’s (Chen & Guestrin, 2016) abstract and introduction. Identify one engineering optimisation it describes that this chapter’s Trade-offs section does not mention. Explain what problem it solves.
12.9 Mental Model
A Random Forest asks: what do a hundred independent experts, each seeing a slightly different version of the data, collectively think? Gradient boosting asks: what’s the smallest correction I can make to my current best guess? Stacking asks: which expert is right in which situation? But it still has to live with whichever experts were invited into the room. That’s why including a much weaker expert, as this chapter’s Worked Example did, can make stacking lose to its own best individual member.