◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Chapter 5 quiz · 25 questions

Information Theory Basics

Tests whether you can compute entropy, cross-entropy, KL divergence, mutual information and information gain by hand, in bits and nats, and spot the traps around zeros, clipping and asymmetry.

Based on chapter 5. Every answer comes with an explanation and a link back to the section.

/Q5

Quiz

Question 1 of 25

Self-information and entropy (bits and nats) Level 1
Progress

An event has probability p=1/8p = 1/8. What is its self-information I(p)=−log⁡2pI(p) = -\log_2 p? Give your answer in bits as an integer.

bits
/02

练习From the book

8 exercises

The chapter's own exercises are open questions. Try one, then compare with the model answer.

Exercise 5.1 [W]

Without computing anything, rank a fair four-sided die, a die with probabilities [0.70,0.10,0.10,0.10][0.70, 0.10, 0.10, 0.10], and a die that always lands on the same face, from highest to lowest entropy, and justify the ranking in one sentence.

Show the model answer

Highest to lowest entropy: fair die (H=2.0000H=2.0000 bits) > biased die [0.70,0.10,0.10,0.10][0.70,0.10,0.10,0.10] (H=1.3568H=1.3568 bits) > constant die (H=0.0000H=0.0000 bits). Entropy tracks how spread-out a distribution is. The fair die is maximally unpredictable: all four outcomes are equally likely. The biased die is more predictable, because one outcome dominates. The constant die has zero uncertainty, because the outcome is already known.

Exercise 5.2 [W]

Explain in your own words why KL(P∥Q)≠KL(Q∥P)\mathrm{KL}(P\|Q) \neq \mathrm{KL}(Q\|P) in general. Give a one-sentence rule for keeping "truth" and "approximation" straight when calling a KL divergence function.

Show the model answer

KL(P∥Q)\mathrm{KL}(P\|Q) measures the expected extra cost of using QQ when PP is the true distribution, weighted by how probable each event is under PP. Swapping PP and QQ changes which distribution's probabilities do the weighting, so the sums aren't the same. The rule for keeping it straight: the first argument is always "truth" (the actual/reference distribution), the second is always the "approximation" being scored against it.

Exercise 5.3 [A]

Using Listing 5.2, compute the binary cross-entropy loss for y_true = [1, 1, 0, 0] against two prediction sets: y_pred_a = [0.6, 0.6, 0.4, 0.4] and y_pred_b = [0.99, 0.51, 0.49, 0.01]. Both models get every prediction "on the correct side of 0.5." Explain why their losses differ anyway.

Show the model answer

Both models classify every example correctly at the 0.5 threshold, but their losses differ because cross-entropy scores confidence calibration, not just which side of the threshold a prediction lands on. y_pred_a = [0.6, 0.6, 0.4, 0.4] is mildly confident about each correct call. Every term is −log⁡(0.6)≈0.5108-\log(0.6) \approx 0.5108 nats, so the average loss is exactly 0.51080.5108. y_pred_b = [0.99, 0.51, 0.49, 0.01] is a mix: two predictions are barely on the correct side (−log⁡(0.51)≈0.6733-\log(0.51) \approx 0.6733 nats each, close to the −log⁡(0.5)≈0.6931-\log(0.5) \approx 0.6931 coin-flip baseline), and two are extremely confident and correct (−log⁡(0.99)≈0.0101-\log(0.99) \approx 0.0101 nats each). Averaging all four gives (0.6733+0.6733+0.0101+0.0101)/4≈0.3417(0.6733 + 0.6733 + 0.0101 + 0.0101)/4 \approx 0.3417 nats: noticeably lower than model A's 0.51080.5108, not similar or higher. The two barely-confident predictions cost close to the coin-flip baseline each, but the two extremely confident predictions cost almost nothing. Cross-entropy's cost for a correct call approaches zero as confidence approaches 1, with no floor above zero. That asymmetry means a model with some very confident correct calls can post a lower average loss than a model that plays it uniformly safe, even while carrying a couple of near-toss-up calls. The lesson: 100% classification accuracy at a threshold says nothing about how good the underlying probabilities are. Cross-entropy does, and it doesn't always penalise a mixed-confidence model more than a uniformly moderate one.

Exercise 5.4 [A]

Using Listing 5.5's pattern, compute the entropy of a five-outcome distribution [0.4,0.4,0.1,0.05,0.05][0.4, 0.4, 0.1, 0.05, 0.05] in bits. State how far it falls below the five-outcome maximum of log⁡2(5)\log_2(5).

Show the model answer

The entropy works out as

H([0.4,0.4,0.1,0.05,0.05])=−(2×0.4log⁡20.4+0.1log⁡20.1+2×0.05log⁡20.05)≈1.8219 bits.\begin{aligned} H([0.4, 0.4, 0.1, 0.05, 0.05]) &= -\left(2\times 0.4\log_2 0.4 + 0.1\log_2 0.1\right. \\ &\qquad \left. {}+ 2\times 0.05\log_2 0.05\right) \\ &\approx 1.8219 \text{ bits.} \end{aligned}

The five-outcome maximum is log⁡2(5)≈2.3219\log_2(5) \approx 2.3219 bits, so this distribution sits exactly 0.50000.5000 bits below maximum entropy. It's noticeably more uncertain than the biased four-sided die in the chapter, since probability mass is still split fairly evenly across the top two outcomes.

Exercise 5.5 [A]

Reproduce Listing 5.7's identity check using P=[0.6,0.4]P = [0.6, 0.4] and Q=[0.5,0.5]Q = [0.5, 0.5], and report KL(P∥Q)\mathrm{KL}(P\|Q), KL(Q∥P)\mathrm{KL}(Q\|P), and whether H(P,Q)=H(P)+KL(P∥Q)H(P,Q) = H(P) + \mathrm{KL}(P\|Q) still holds.

Show the model answer

For P=[0.6,0.4]P=[0.6,0.4], Q=[0.5,0.5]Q=[0.5,0.5]: KL(P∥Q)=0.6log⁡2(0.6/0.5)+0.4log⁡2(0.4/0.5)≈0.6×0.2630+0.4×(−0.3219)≈0.1578−0.1288=0.0290\mathrm{KL}(P\|Q) = 0.6\log_2(0.6/0.5) + 0.4\log_2(0.4/0.5) \approx 0.6\times0.2630 + 0.4\times(-0.3219) \approx 0.1578 - 0.1288 = 0.0290 bits. KL(Q∥P)=0.5log⁡2(0.5/0.6)+0.5log⁡2(0.5/0.4)≈0.5×(−0.2630)+0.5×0.3219≈−0.13152+0.16096=0.02945≈0.0294\mathrm{KL}(Q\|P) = 0.5\log_2(0.5/0.6) + 0.5\log_2(0.5/0.4) \approx 0.5\times(-0.2630) + 0.5\times 0.3219 \approx -0.13152+0.16096 = 0.02945 \approx 0.0294 bits. That's close to but not equal to KL(P∥Q)\mathrm{KL}(P\|Q), confirming the asymmetry again. The two happen to be numerically close here only because PP and QQ are close to each other. H(P)=−(0.6log⁡20.6+0.4log⁡20.4)≈0.9710H(P) = -\left(0.6\log_2 0.6 + 0.4\log_2 0.4\right) \approx 0.9710 bits. H(P)+KL(P∥Q)≈0.9710+0.0290=1.0000H(P) + \mathrm{KL}(P\|Q) \approx 0.9710 + 0.0290 = 1.0000 bits. That matches H(P,Q)=−(0.6log⁡20.5+0.4log⁡20.5)=1.0000H(P,Q) = -(0.6\log_2 0.5 + 0.4\log_2 0.5) = 1.0000 bits exactly. Since QQ is uniform over two outcomes, H(P,Q)H(P,Q) reduces to exactly 1 bit regardless of PP. The identity holds.

Exercise 5.6 [E]

A colleague proposes ranking every feature in a 200-column dataset by mutual information against the label and keeping only those above a score of 0.1, arguing "0.1 is a reasonable bar, like a correlation of 0.1." Using this chapter's Trade-offs section, explain what's wrong with treating 0.1 as a fixed, portable threshold.

Show the model answer

A correlation of 0.1 and a mutual-information score of 0.1 are not on the same scale, and don't mean the same thing. Correlation is bounded in [−1,1][-1,1], with well-understood conventions for "weak." Mutual information, in nats, as mutual_info_classif returns it, has no fixed scale. Its ceiling is the label's entropy, and its magnitude depends on the entropy of the specific label and feature distributions in that dataset. A score of 0.1 might be near the noise floor for one problem and a strong, useful signal for another, as the Worked Example showed with feature_noise scoring 0.043 against feature_nonlinear's 0.338 on the same dataset. The chapter's Trade-offs section is explicit that MI should only be used to rank features within a single run, not compared against a fixed external threshold. The right move is to always rank features against each other (or against a known-noise baseline feature computed on the same data) rather than picking an absolute cutoff.

Exercise 5.7 [E]

Design a synthetic feature, distinct from feature_nonlinear in the Worked Example, whose mean is unrelated to a binary label but that is still informative about it (i.e., correlation should be near zero but mutual information should be meaningfully above the noise floor). Describe the mechanism in words. You don't need to run it.

Show the model answer

Model answer: a binary label determines which of two shapes a numeric feature is drawn from, both centred at zero. For example, when label is 0, the feature is drawn from a standard normal. When label is 1, the feature is drawn from a mixture that is symmetric around zero but bimodal (e.g., a 50/50 mix of N(−2,0.3)\mathcal{N}(-2, 0.3) and N(2,0.3)\mathcal{N}(2, 0.3)). The mean of the feature is approximately zero regardless of the label, so correlation with the label is near zero. But the shape of the distribution, unimodal versus bimodal, differs sharply between labels. That's exactly the kind of higher-order dependence mutual information, but not correlation, is sensitive to. This is a generalisation of the Worked Example's variance-based trick to an even more clearly non-linear signal (shape rather than spread).

Exercise 5.8 [R]

Read Shannon's original 1948 paper, or at minimum its introduction. Identify one idea in it that this chapter did not cover, and explain in a few sentences how it relates to (or extends) the entropy/cross-entropy/KL/mutual-information picture built here.

Show the model answer

Discussion only. No single answer. Shannon's paper introduces channel capacity and the noisy-channel coding theorem, which this chapter never touches. The idea: a communication channel has a maximum rate at which information can be transmitted with arbitrarily low error, and entropy sets the fundamental limit on lossless compression. Worth considering how this connects to the chapter's ML framing: a classifier's cross-entropy loss floor at H(P)H(P) is structurally the same idea as a channel's capacity limit. No amount of cleverness can push performance past a limit set by the intrinsic uncertainty of the source. Students should also notice that Shannon's original motivation was communication engineering, not statistics or machine learning, which came decades later and inherited the same mathematics.

online /books/classical-data-science/quiz/5 Newcastle, UK v0.1.0 // rss