Simpson's Paradox

A treatment works better than the alternative for men. It also works better for women. And it works worse for the combined population. This is Simpson's paradox, and it is not a trick: the arithmetic is elementary and the reversal is entirely real.

Its philosophical interest is that it establishes a negative result of considerable generality. No purely statistical rule determines which set of probabilities is the one to act on. The aggregated and partitioned data are both correct; they answer different questions; and deciding which question is the relevant one requires knowledge that is not contained in any distribution.

The reversal

The kidney-stone data of Charig et al. is the standard example.

	Treatment A	Treatment B
Small stones	93% (81/87)	87% (234/270)
Large stones	73% (192/263)	69% (55/80)
Overall	78% (273/350)	83% (289/350)

Treatment A is better for small stones and better for large stones, and worse overall. The explanation is that the groups are not comparable: treatment A was given predominantly to large-stone cases, which have worse outcomes regardless of treatment. Stone size is a confounder — it affects both which treatment was assigned and the outcome — and the aggregate comparison mixes the treatment effect with the effect of the case mix.

Formally, the reversal is possible because

$\frac{a _{1} + a _{2}}{b _{1} + b _{2}} is not determined by \frac{a _{1}}{b _{1}} and \frac{a _{2}}{b _{2}}$

Ratios do not aggregate by averaging; the weights matter. There is nothing paradoxical in the mathematics. The paradox is entirely in the expectation that a relation holding in every subgroup must hold overall — an expectation that would be correct for many relations but is not for conditional probabilities.

The sure-thing principle

Savage's sure-thing principle says: if you would prefer $A$ to $B$ knowing that $C$ , and would also prefer $A$ to $B$ knowing that $\neg C$ , then you should prefer $A$ to $B$ not knowing whether $C$ . It looks unimpeachable.

Simpson's paradox appears to refute it, and the resolution is worth being precise about because it identifies exactly what goes wrong. The principle is valid when the partition is independent of the choice. It fails when learning the outcome would change what you know about which condition obtains — or, in the causal version, when the act itself influences the partition or is correlated with it.

In the kidney-stone case the partition is not independent of treatment assignment: doctors assigned treatment A to the harder cases. So the conditions of the principle are not met, and there is no genuine counterexample. The lesson generalises: the aggregate comparison is uninformative about the effect of choosing a treatment precisely because the observed assignment was not a choice made independently of the patient's condition.

Which partition?

The obvious response is that one should always condition on more variables. This is wrong, and seeing why is the philosophical heart of the matter.

Consider a drug that affects recovery by lowering blood pressure. Partitioning by post-treatment blood pressure will make the drug appear ineffective within each stratum, because conditioning on a mediator removes the very pathway through which the drug works. Here the aggregate figure is the correct one and the partitioned figure is misleading — the exact reverse of the kidney-stone case.

So the data can reverse under partition, and the correct answer is sometimes the aggregate and sometimes the strata. The statistics are identical in structure in the two cases. What differs is the causal role of the partitioning variable:

Confounder (a common cause of treatment and outcome) — condition on it. Stone size influences both which treatment was given and the outcome.
Mediator (on the causal path from treatment to outcome) — do not condition on it. Blood pressure is how the drug acts.
Collider (a common effect of treatment and outcome) — do not condition on it; doing so creates spurious association.

No feature of the joint distribution distinguishes these three. Two data sets can be numerically identical while the correct analysis differs, because the causal structures differ. This is the sharpest available demonstration that causal conclusions require causal assumptions, and that they cannot be extracted from statistics alone however sophisticated the technique.

The formal treatment is Pearl's: the question "what is the effect of the treatment?" concerns $P (recovery ∣ d o (treatment))$ , which is not in general equal to $P (recovery ∣ treatment)$ , and computing it requires a causal model specifying which variables are confounders. The back-door criterion then says precisely which sets of variables to condition on.

Real cases

Berkeley admissions (1973). Aggregate data showed women admitted at a lower rate than men, suggesting discrimination. Department by department, women were admitted at slightly higher rates in most departments. Women applied disproportionately to departments with low admission rates overall.

This case is worth dwelling on, because it shows that the choice of analysis is not always resolvable by getting the causal graph right. Department is a mediator on one causal pathway from gender to admission — if women applied to more competitive departments because of prior discouragement, or because of how fields are gendered, then conditioning on department removes part of the effect of interest. Whether to condition depends on whether "discrimination" means differential treatment of comparable applicants at the point of decision, or the total effect of gender on admission probability. Those are different questions and they have different answers. The statistics cannot choose between them, and neither can causal modelling alone: it requires deciding what the question is.

Batting averages, wage gaps, and mortality rates exhibit the same structure routinely, and the same interpretive choice arises in each.

Where this sits

Simpson's paradox is the most compact argument in this section for the conclusion that probability underdetermines causal structure. The numbers are consistent, the arithmetic elementary, and the correct action undetermined by the distribution.

Three morals carry forward. First, conditional probabilities are not effects: $P (E ∣ C)$ answers a question about observation, and intervention requires a different quantity. Second, more conditioning is not better: conditioning on mediators and colliders introduces bias rather than removing it. Third, the aggregate/partition choice is a causal question, and no amount of data settles it.

These are exactly the points the causal models framework formalises, which is the next page. It supplies the notation for distinguishing observing from intervening, and a criterion for deciding what to condition on.