Chapter 5
Information Theory Basics
Why does classification use a logarithm in its loss function? Why do decision trees split on “information gain”? Why does a variational autoencoder’s loss include a strange penalty term called KL divergence? Why does RLHF constrain a fine-tuned model with something by the same name? All four questions collapse to a single answer: information theory. The mathematics of uncertainty. It gives a precise vocabulary for how much a distribution doesn’t know, how expensive a wrong prediction is, and how far apart two probability distributions sit from each other. This chapter builds that vocabulary from a single equation up through four concepts (entropy, cross-entropy, KL divergence, and mutual information) that recur constantly for the rest of this book, usually without ever being named.
5.1 Motivation
Every concept in this chapter comes from one idea: how “surprising” is an event? Flipping a fair coin and getting heads is unsurprising. Flipping an unfair coin that almost always lands heads and getting tails is very surprising. Claude Shannon formalised this in 1948 with the information content (or self-information) of an event with probability :
Low-probability events carry high information content: they’re surprising. High-probability events carry low information content: they’re expected. Entropy is the average of this quantity over a whole distribution. Cross-entropy and KL divergence compare two distributions using the same log-probability machinery. Mutual information asks how much one variable’s distribution shifts when another is known. This is one idea, viewed from four angles. Those four angles happen to be exactly the four things a data scientist needs when training and evaluating models.
5.2 Exposition
5.2.1 Entropy: How Uncertain Is a Distribution?
Entropy, written , is the average information content of a distribution. For a discrete distribution over outcomes:
Entropy measures uncertainty, not correctness. It says nothing about whether a distribution is right, only how spread out it is. Bounded by two extremes. At maximum entropy, the distribution is uniform: with classes, . There’s no information favouring any one outcome over another. At minimum entropy, : a point mass, one outcome with probability 1. Zero uncertainty.
This is why decision trees split on entropy (or the closely related Gini impurity, discussed below). At a tree node, entropy measures how mixed the classes are. A good split is one that produces child nodes with lower combined entropy than the parent. Those children are more predictable, closer to a point mass, than the node they came from. The information gain of a split is the entropy reduction it gives.
Gini impurity measures the same thing entropy does, how mixed a set of classes is, but with a different formula:
Picture picking one example at random from a node and labelling it with a random guess drawn from the node’s own class proportions. Gini impurity is the probability that guess is wrong. A pure node (every example the same class) has Gini impurity 0. Nobody could guess wrong there. A node split evenly across two classes has Gini impurity 0.5, a coin flip. Like entropy, it bottoms out at a point mass and peaks when classes are evenly mixed. The Trade-offs section below compares the two.
5.2.2 Cross-Entropy: The Cost of Being Wrong
Cross-entropy measures how well a predicted distribution approximates a true distribution :
When (the model predicts exactly right), cross-entropy equals entropy, its minimum possible value. When diverges from , cross-entropy grows. It grows in a specific, asymmetric way: the model pays a small penalty for being uncertain, and a large penalty for being confidently wrong.
This is the loss function used to train classifiers. In binary classification, is the true label (0 or 1) and is the model’s predicted probability . The binary cross-entropy loss is:
When is close to , is small. When is far from (say but ), is a large negative number and explodes. Cross-entropy punishes confident wrong predictions far more heavily than uncertain ones. That’s the behaviour wanted from a classification loss: a model that says “I’m 99% sure this is fraud” and is wrong should be penalised much harder than one that shrugged and said “meh, maybe, 50/50.”
Cross-entropy and entropy are linked by an identity that matters more than you might think (if you’re new to this, you’d have no reason to think it matters at all):
Cross-entropy equals the true entropy plus a penalty term (KL divergence) for using the wrong distribution. , the entropy of the true labels, is fixed. It doesn’t depend on the model’s parameters. So minimising cross-entropy during training is mathematically identical to minimising KL divergence between the model’s predictions and the truth.
5.2.3 KL Divergence: How Far Is Q From P?
KL divergence (Kullback-Leibler divergence) is a directed measure of how different a distribution is from a reference distribution :
Three properties matter in practice. First, always, with equality only when . It behaves somewhat like a distance. Second, and unlike a true distance, it is not symmetric: in general. It’s a directed divergence, not a metric. Third, it is undefined (infinite) at any point where but . must cover the support of , or the divergence blows up.
KL divergence shows up constantly beyond the classification loss it’s hiding inside. A variational autoencoder’s loss includes a KL term that pulls the learned latent distribution towards something close to a standard Gaussian. RLHF constrains a fine-tuned language model with a KL penalty against the base model. That keeps optimisation for the reward signal from drifting the model into degenerate, reward-hacking behaviour. Knowledge distillation trains a small student model to minimise its KL divergence from a large teacher model’s output distribution. TRPO, and the KL-penalty variant of PPO, use KL divergence between successive policies to keep updates from being too large. In every one of these, the same asymmetric, always-non-negative quantity from the equation above is doing the work.
A latent variable is one the model never observes directly. It’s the model’s own internal, compressed summary of the input, learned rather than measured. A variational autoencoder compresses each input (an image, say) down to a small vector of numbers, its latent representation. It then tries to reconstruct the original input from just that vector. The latent distribution is the probability distribution over those compressed vectors that the model learns to produce. Pulling it towards a standard Gaussian, via the KL term mentioned above, keeps that compressed space well-behaved: smooth, centred, and usable for generating new examples by sampling from it later. Book 2 covers variational autoencoders and their architecture in full.
5.2.4 Mutual Information: How Much Does Knowing X Tell You About Y?
Mutual information measures how much knowing one variable reduces uncertainty about another:
Equivalently, : mutual information is the KL divergence between the actual joint distribution of and and the joint distribution they would have if independent. When , and are independent: knowing one tells you nothing about the other. When is large, they’re strongly dependent.
The practical use is feature selection:
sklearn.feature_selection.mutual_info_classif ranks candidate features by how much each reduces uncertainty about the label. Chapter 3 built correlation and covariance out of sums of products of deviations from the mean. That machinery only ever sees straight-line relationships. Mutual information asks the same practical question, does this feature relate to the label, but answers it from entropy instead of deviations. That different foundation is exactly why it succeeds where correlation fails. Nothing in its construction assumes the relationship is linear, so it catches dependence covariance is structurally blind to. Mutual information also underlies the information bottleneck framework, for understanding what a neural network’s internal layers retain. It is also behind contrastive methods like SimCLR and CLIP. Their InfoNCE loss maximises a lower bound on the mutual information between paired inputs: two augmented views of one image (SimCLR), or an image and its caption (CLIP).
5.2.5 How the Four Fit Together
Entropy is the baseline difficulty of a problem. Cross-entropy is what a model is being scored on. KL divergence is the gap between the two. Mutual information asks the same underlying question (how much does one thing tell you about another) of two variables rather than of a model against the truth.
5.3 Trade-offs and Limits
Entropy and Gini impurity are the two candidates for measuring node impurity in decision trees. In practice they produce nearly identical splits. Entropy is more theoretically grounded: it’s the same quantity used everywhere else in this chapter. Gini is a little cheaper to compute, since it needs no logarithm, and scikit-learn defaults to it. The choice rarely changes which feature a tree picks.
Cross-entropy versus mean squared error for classification is not such a close call. Cross-entropy is the mathematically correct loss: it falls directly out of the likelihood of a Bernoulli or categorical distribution. MSE applied to classification produces saturated gradients near 0 and 1, exactly where a classifier most needs strong gradient signal to fix a confident, wrong prediction. Cross-entropy should always be used for classification. Reaching for MSE there is a common and avoidable mistake.
KL divergence’s asymmetry could be seen as a flaw. Better to treat it as a feature to be deliberate about. KL(P||Q) and KL(Q||P) answer different questions. Mixing them up produces subtly wrong regularisation or drift-detection behaviour. Always be explicit about which distribution is “truth” (the first argument) and which is the “approximation” (the second). Mutual information’s main limitation is that, unlike a correlation coefficient’s fixed range, it has no fixed scale: its ceiling is the label’s entropy, which changes from dataset to dataset, and there’s no universal “this counts as strong” threshold. It is safe to use for ranking features against each other within one dataset and one run, but not as an absolute quality score comparable across different problems.
5.4 Procedures
Procedure 5.1 (Compute Entropy of a Distribution). When to use: summarising how uncertain or spread out a distribution is: class balance in a dataset, or a model’s predicted class probabilities.
import numpy as np
import scipy.stats as stats
def entropy(p, eps=1e-12):
p = np.clip(np.asarray(p, dtype=float), eps, 1.0)
return -np.sum(p * np.log2(p)) # bits
# Imbalanced binary label
class_balance = np.array([0.9, 0.1])
# Low -- easy to predict the majority class
print(entropy(class_balance))
# scipy equivalent (natural log by default -- pass base=2
# to match bits)
print(stats.entropy(class_balance, base=2))Common mistake: passing raw counts instead of a normalised distribution. entropy([90, 10]) is wrong. Normalise first with p / p.sum(), or let scipy.stats.entropy do it (it normalises automatically, unlike a hand-rolled version).
Procedure 5.2 (Compute Cross-Entropy Loss). When to use: training or evaluating a classifier, or sanity-checking predicted probabilities against ground truth.
import numpy as np
from sklearn.metrics import log_loss
def binary_cross_entropy(y_true, y_pred, eps=1e-12):
y_pred = np.clip(y_pred, eps, 1 - eps)
return -np.mean(
y_true * np.log(y_pred)
+ (1 - y_true) * np.log(1 - y_pred)
)
y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.2, 0.6, 0.95])
print(binary_cross_entropy(y_true, y_pred))
# sklearn handles the multi-class case and the clipping for you
print(log_loss(y_true, y_pred))Here is a rule of thumb for reading raw loss values (natural log, nats). Around 0.69 is no better than a coin flip on a balanced binary problem (); below 0.1 is confident and mostly correct; above 2.0 means the model is confidently wrong on a meaningful fraction of examples.
Common mistake: feeding raw logits into a loss function that expects probabilities, or vice versa. sklearn.metrics.log_loss expects probabilities in . PyTorch’s nn.CrossEntropyLoss expects raw logits and applies softmax internally. Passing already-softmaxed probabilities double-applies the transform and silently produces a plausible-looking but wrong loss.
Procedure 5.3 (Compute KL Divergence Between Two Distributions). When to use: comparing a predicted distribution against a reference or prior, checking a feature’s distribution for drift over time, or implementing a VAE or RLHF penalty term.
import numpy as np
import scipy.stats as stats
def kl_divergence(p, q, eps=1e-12):
p = np.clip(np.asarray(p, dtype=float), eps, 1.0)
q = np.clip(np.asarray(q, dtype=float), eps, 1.0)
return np.sum(p * np.log2(p / q))
p = np.array([0.5, 0.3, 0.2])
q = np.array([0.3, 0.3, 0.4])
print(kl_divergence(p, q)) # KL(P || Q)
# KL(Q || P) -- different! not symmetric
print(kl_divergence(q, p))
# scipy: entropy(p, q) computes KL(p || q) when q is provided
print(stats.entropy(p, q, base=2))Common mistake: treating KL divergence as an interchangeable distance metric and swapping the arguments freely. KL(P||Q) != KL(Q||P), so always be explicit about which is truth (first argument) and which is approximation (second). Separately, KL divergence is infinite wherever q is zero and p isn’t, which is the practical reason for the eps clipping above.
Procedure 5.4 (Rank Features by Mutual Information). When to use: selecting predictive features when a non-linear relationship to the label is suspected, one that a correlation matrix would miss entirely.
from sklearn.feature_selection import (
mutual_info_classif,
mutual_info_regression,
)
import pandas as pd
# Classification target
mi_scores = mutual_info_classif(X, y, random_state=42)
ranking = (
pd.Series(mi_scores, index=X.columns)
.sort_values(ascending=False)
)
print(ranking)
# Regression target
mi_scores_reg = mutual_info_regression(
X, y_continuous, random_state=42
)
# Keep only the top-K informative features
top_k_features = ranking.head(10).index
X_selected = X[top_k_features]Common mistake: comparing mutual information scores across different runs or datasets as if they sat on a fixed, interpretable scale like a correlation’s range. MI is non-negative, but it has no fixed scale. For a class label, its ceiling is the label’s entropy, which changes from dataset to dataset. Use it only to rank features relative to each other within the same run, not as an absolute quality threshold.
Procedure 5.5 (Diagnose a Cross-Entropy Loss That Won’t Go Down). When to use: training a classifier whose loss is stuck, exploding, or has gone to nan.
| Symptom | Most likely cause | Fix |
|---|---|---|
Loss is nan |
log(0) from an unclipped prediction of exactly 0 or 1 |
Clip predictions to , or use a numerically stable framework loss instead of a manual log |
| Loss stuck near | Model is predicting close to uniform, hasn’t learned anything yet | Check the learning rate. Check labels aren’t shuffled or misaligned |
| Loss decreases, then suddenly spikes | An extreme confident-and-wrong prediction on a mislabelled or outlier example | Inspect the highest per-example losses. Check for label noise |
| Train loss low, validation loss high | Overfitting, not an information-theory problem | Regularisation, covered later in this book |
5.5 Worked Example
The companion notebook implements all four quantities from scratch in numpy and checks that they behave the way the theory in this chapter predicts, rather than deriving anything new.
It starts with entropy over a four-sided die in three configurations, using Listing 5.1’s function:
fair_die = np.array([0.25, 0.25, 0.25, 0.25])
biased_die = np.array([0.70, 0.10, 0.10, 0.10])
point_mass = np.array([1.00, 0.00, 0.00, 0.00])
distributions = [
('Fair die', fair_die),
('Biased die', biased_die),
('Point mass', point_mass),
]
for name, dist in distributions:
print(f"{name:12s}: H = {entropy(dist):.4f} bits")The computed results: the fair die has bits, exactly , the maximum possible for four outcomes. The biased die has bits, less uncertainty, since one outcome dominates. The point mass has bits, nothing left to be uncertain about. All three land exactly where the Exposition section’s theory says they should.
Next, the notebook simulates a single true label and computes binary cross-entropy at three predicted probabilities:
y_true_single = np.array([1.0])
l_correct = binary_cross_entropy(y_true_single, np.array([0.99]))
l_unsure = binary_cross_entropy(y_true_single, np.array([0.50]))
l_wrong = binary_cross_entropy(y_true_single, np.array([0.01]))
print(f"Confident and correct (ŷ=0.99): loss = {l_correct:.4f}")
print(f"Uncertain (ŷ=0.50): loss = {l_unsure:.4f}")
print(f"Confident and WRONG (ŷ=0.01): loss = {l_wrong:.4f}")The computed losses are nats for the confident-correct prediction, nats for the maximally uncertain one, and nats for the confident-and-wrong one. The confidently wrong prediction costs 458 times as much as the confident-correct one, and about 6.6 times as much as the maximally uncertain one. That’s a sharply asymmetric penalty. It’s exactly the behaviour that makes cross-entropy, not MSE, the correct loss for classification.
The notebook then verifies the identity directly, using and :
P = np.array([0.50, 0.30, 0.15, 0.05])
Q = np.array([0.30, 0.30, 0.20, 0.20])
kl_pq = kl_divergence(P, Q)
kl_qp = kl_divergence(Q, P)
h_p = entropy(P)
h_pq = cross_entropy_dist(P, Q)The computed values: bits and bits. That confirms the asymmetry claimed in the Exposition. bits. bits. That matches the directly computed bits exactly. The identity linking cross-entropy, entropy, and KL divergence isn’t an approximation. It holds bit for bit.
Finally, the notebook builds a small synthetic dataset designed to expose correlation’s blind spot: feature_linear, whose mean depends on the label; feature_nonlinear, whose variance (not mean) depends on the label; and feature_noise, unrelated to either. Ranking all three by both and mutual information, using Listing 15.3, produces:
| Feature | |correlation| | Mutual information (nats) |
|---|---|---|
feature_nonlinear |
0.0161 | 0.3377 |
feature_linear |
0.7047 | 0.3303 |
feature_noise |
0.0075 | 0.0430 |
Correlation ranks feature_nonlinear (0.0161) as statistically indistinguishable from pure noise (0.0075). Its mean doesn’t move with the label, only its spread does, and Pearson correlation is blind to that. Mutual information ranks it as the single most informative feature in the dataset, fractionally ahead of feature_linear. It measures the full distributional dependence between feature and label, not just a linear one. This is the practical payoff of everything in this chapter’s Exposition section: mutual information catches exactly the relationship correlation is mathematically incapable of seeing.
This chapter has no AWS SageMaker Notes sidebar. Every computation above runs on CPU in milliseconds. For mutual-information feature selection at a scale that no longer fits comfortably in a notebook, mutual_info_classif can run inside a SageMaker Processing job. That pattern reappears in this book once a chapter has a dataset large enough to need it.
5.6 History and Further Reading
Murphy’s Introduction, Ch. 6, gives the full treatment of entropy, cross-entropy, KL divergence, mutual information, and their connection to data compression. It’s the natural next stop after this chapter (Murphy 2022). Burkov covers cross-entropy specifically as a loss function, concisely and directly tied to the classification setting this chapter used to motivate it (Burkov 2019, 34–40). Shannon’s original 1948 paper is the source of essentially everything in this chapter. It’s unusually readable for a foundational paper of this significance (Shannon 1948). Christopher Olah’s essay “Visual Information Theory” (colah.github.io, 2015) is, informally, the best available visual intuition for entropy, cross-entropy, and KL divergence. It’s worth reading alongside this chapter’s equations, rather than instead of them.
5.7 How This Connects
This chapter builds on Chapter 3’s treatment of probability and distributions. Information content is only meaningful once “probability of an event” is already a settled idea. It enables the loss functions behind decision trees and neural network training (Chapter 11 and later chapters on classical and deep models). It also underpins variational autoencoders and RLHF (covered in Book 2) and the feature-selection techniques that recur throughout the data preparation chapters of this book. It is most often confused with a mix-up between entropy and cross-entropy. Entropy measures the uncertainty of the true distribution alone. Cross-entropy measures the cost of using a predicted distribution in its place. They differ by exactly the KL divergence between the two. Training a classifier means minimising cross-entropy. That indirectly minimises the KL term, since entropy is fixed.
5.8 Exercises
[W] Without computing anything, rank a fair four-sided die, a die with probabilities , and a die that always lands on the same face, from highest to lowest entropy, and justify the ranking in one sentence.
[W] Explain in your own words why in general. Give a one-sentence rule for keeping “truth” and “approximation” straight when calling a KL divergence function.
[A] Using Listing 5.2, compute the binary cross-entropy loss for
y_true = [1, 1, 0, 0]against two prediction sets:y_pred_a = [0.6, 0.6, 0.4, 0.4]andy_pred_b = [0.99, 0.51, 0.49, 0.01]. Both models get every prediction “on the correct side of 0.5.” Explain why their losses differ anyway.[A] Using Listing 5.5’s pattern, compute the entropy of a five-outcome distribution in bits. State how far it falls below the five-outcome maximum of .
[A] Reproduce Listing 5.7’s identity check using and , and report , , and whether still holds.
[E] A colleague proposes ranking every feature in a 200-column dataset by mutual information against the label and keeping only those above a score of 0.1, arguing “0.1 is a reasonable bar, like a correlation of 0.1.” Using this chapter’s Trade-offs section, explain what’s wrong with treating 0.1 as a fixed, portable threshold.
[E] Design a synthetic feature, distinct from
feature_nonlinearin the Worked Example, whose mean is unrelated to a binary label but that is still informative about it (i.e., correlation should be near zero but mutual information should be meaningfully above the noise floor). Describe the mechanism in words. You don’t need to run it.[R] Read Shannon’s original 1948 paper, or at minimum its introduction. Identify one idea in it that this chapter did not cover, and explain in a few sentences how it relates to (or extends) the entropy/cross-entropy/KL/mutual-information picture built here.
5.9 Mental Model
Entropy is the baseline: how hard is this problem, before any model touches it? Cross-entropy is the scorecard: how well is a model doing against that baseline? KL divergence is the gap between the two: how much worse than optimal is the model being? Mutual information asks the same question of two variables instead of a model and the truth: how much does one tell you about the other? When a classifier’s cross-entropy loss converges to , the model has learned everything there is to learn from the data. No amount of further training can close a gap that no longer exists.