◢◤ ARKANE_
SYS_TIME UTC+0
Work with me

Chapter 7 quiz · 24 questions

Data Cleaning & Wrangling

Tests whether you can diagnose and treat missing values, outliers, duplicates, wrong types, encoding errors and messy categories, and predict what the pandas cleaning code does.

Based on chapter 7. Every answer comes with an explanation and a link back to the section.

/Q7

Quiz

Question 1 of 24

Missing-value mechanisms and imputation Level 1
Progress

Missingness that depends on observed variables, but not on the missing value itself, is called what?

Choose one
/02

练习From the book

7 exercises

The chapter's own exercises are open questions. Try one, then compare with the model answer.

Exercise 7.1 [W]

Classify each of the following as MCAR, MAR, or MNAR, and justify your answer in one sentence: (a) a temperature sensor that randomly drops 2% of readings due to a hardware fault; (b) a salary survey where respondents earning over $500k are half as likely to answer the income question as everyone else; (c) a customer age field where missingness is higher among users who signed up via a mobile app that didn't require an age field, regardless of their actual age.

Show the model answer

(a) MCAR: a hardware fault dropping readings at random has no relationship to the reading's value or any other variable. (b) MNAR: the probability of a missing income value depends directly on the (unobserved) income itself, higher earners are more likely to decline. (c) MAR: missingness correlates with an observed variable (signup channel), not with the true age value itself; conditional on signup channel, the missing ages aren't systematically different from the reported ones.

Exercise 7.2 [W]

Using Listing 7.8's logic, explain why ages[0] = 312 is assigned after the missing-value assignment ages[missing_age_idx] = np.nan. Explain what would change about row 0 if the two lines were swapped.

Show the model answer

The typo assignment ages[0] = 312 is applied after the missing-value assignment, so that it deterministically overwrites row 0, regardless of whether index 0 happened to be selected by np.random.choice(n, 40, replace=False). If the two lines were swapped, row 0 would first be set to 312. Then, only if index 0 were among the 40 randomly chosen missing indices, it would get overwritten back to NaN. That would make the presence of the planted outlier depend on random chance, rather than being guaranteed. The chapter's ordering guarantees the outlier exists every time the notebook runs with a fixed seed.

Exercise 7.3 [A]

Using Listing 7.9 as a reference, compute what the age imputation value would need to be if, instead of median imputation, mean imputation were used on the post-capping age column. Describe the one-line change. State whether you'd expect the result to be higher or lower than 42, and why.

Show the model answer

The one-line change is df['age'] = df['age'].fillna(df['age'].mean()) in place of .median(). After deduplication and before capping, age has a mean of 42.82 against a median of 42, the 312 outlier pulling the mean up above the rank-based median. Capping that single value down to 90 drags the mean back down to about 42.34, very close to the median of 42. So mean imputation would land at roughly 42.3, a third of a point above the reported median-imputed value of 42. The small remaining gap comes from the mild right skew still left in age even after the domain cap.

Exercise 7.4 [A]

The chapter reports that IQR-based detection flags only 1 outlier in the raw age column despite there being an obvious typo (312). Using Listing 7.4's iqr_bounds function, explain why a single extreme value like 312 adds only one to the outlier count and barely moves the fence, given how Q1Q_1 and Q3Q_3 are computed.

Show the model answer

Q1 and Q3 are the 25th and 75th percentile of the whole column. Only one value, 312, sits far above the bulk of the data (ages spread across 22–64). That single point has almost no leverage on the rank-based Q1/Q3 statistics. 312 is one point out of hundreds, so the 25th/75th percentile boundaries barely move to accommodate it. The resulting IQR fence lands well below 312, meaning IQR does in fact catch it. That's the reported "1 outlier." The subtler risk is the opposite one: IQR is a whole-column, single-pass method. A handful of outliers or a long natural tail can each be caught or missed, depending on how many other values sit between the bulk of the distribution and the fence. That's why the chapter still applies a domain-knowledge cap ([16, 90]) on top of IQR, rather than trusting IQR's fence alone.

Exercise 7.5 [E]

A colleague proposes skipping validation (Procedure 7.6) on the grounds that "the pipeline already ran without errors, so the data must be fine." Using this chapter's Trade-offs section, write the two or three sentences you'd say to push back, with a concrete example of a silent failure that would pass unnoticed without an explicit validation step.

Show the model answer

A pipeline running without errors only confirms that every operation was syntactically and type-valid. It says nothing about whether the result is semantically correct. A concrete silent failure: an incorrectly specified gender_map that's missing the 'MALE' (all-caps) variant wouldn't raise any error. .map() would convert every 'MALE' entry to NaN, silently turning a known category into missing data. The pipeline would report success right up until someone later asks why 50 rows have a missing gender that used to have a value.

Exercise 7.6 [E]

Design a validation rule set (in the style of Listing 7.6) for a dataset of e-commerce orders with columns order_total, discount_pct, ship_date, delivery_date, and customer_id. Propose at least four rules, including at least one that checks a relationship between two columns rather than a single column in isolation.

Show the model answer

Sample rules: (1) (d['order_total'] > 0).all(), no order should have a non-positive total; (2) (d['discount_pct'] >= 0).all() and (d['discount_pct'] <= 100).all(), a discount percentage must be a valid percentage; (3) d['customer_id'].notnull().all(), every order must be attributable to a customer; (4) a cross-column rule: (d['delivery_date'] >= d['ship_date']).all(), a package cannot be delivered before it ships. Single-column rules can never catch that, since both dates can individually look perfectly valid. A fifth candidate, if the data also had an order_date column: (d['ship_date'] - d['order_date']).dt.days.between(0, 30).all() as a sanity bound on fulfilment time, catching data-entry errors that swap order and ship dates.

Exercise 7.7 [R]

MNAR missingness is described in this chapter as requiring you to "model the missingness mechanism, not just the missing value." Research one concrete technique for this (e.g. Heckman correction, pattern-mixture models, or multiple imputation with a missingness indicator) and summarise, in your own words, what data beyond the column itself it requires.

Show the model answer

Discussion only, no single answer. Multiple imputation with a missingness indicator is the most tractable technique to start with. Alongside the imputed value, an explicit binary column recording whether the value was originally missing is added as a feature. That lets a downstream model learn that "this value was estimated" is itself informative, relevant when missingness correlates with the outcome, as in MNAR. A Heckman correction goes further, formally modelling a "selection equation" for whether a value is observed at all, jointly with the equation of interest. It requires an instrument variable that affects observability but not the outcome directly. That's often the hardest part to find in practice. Worth considering: unlike MCAR/MAR imputation, none of these techniques make the MNAR problem disappear. They make its assumptions explicit and testable rather than silently violated.

online /books/classical-data-science/quiz/7 Newcastle, UK v0.1.0 // rss