Comparing Two Groups

Aspirin, 1988

On September 20, 1982, the Physicians' Health Study assigned its first participant. It was a randomized, double-blind, placebo-controlled trial: 22,071 healthy male doctors in the United States were randomly assigned to take either 325 mg of aspirin every other day or a placebo that looked the same. Nobody, not the doctors taking pills and not the researchers checking them, knew who had which.

The trial was meant to run for years. But on December 18, 1987, its independent monitoring board recommended stopping the aspirin part early because the benefit against heart attacks was already statistically extreme, and January 25, 1988 became its official end. The final report, in the New England Journal of Medicine in 1989, gave these counts after an average of about five years (60.2 months):

The report summed this up as a 44% reduction in risk (relative risk 0.56, 95% confidence interval 0.45 to 0.70, P < 0.00001). That is three questions at once. Is the difference between 1.26% and 2.17% bigger than chance would produce? How big is it, with a margin of error? And how much does it matter to one person deciding whether to take a pill? This unit answers all three.

From one group to two

In Inference every question was about one group: one proportion (one-proportion z test) or one mean (one-sample t test). With two groups the parameter is a difference: p₁ − p₂ or μ₁ − μ₂. The pattern stays the same:

• Interval: estimate ± critical value × standard error.
• Test: (estimate − null value) ÷ standard error, then a p-value.

One new fact does most of the work. For two independent samples, variances add:

SE(difference) = √( SE₁² + SE₂² )

Standard errors do not add, but their squares do. The difference is less precise than either estimate alone, which makes sense: two sources of chance error instead of one. Where a difference comes from also matters. A randomized experiment like the aspirin trial lets a significant difference be read as caused by the treatment; an observational comparison does not (see Sampling and Surveys).

Difference of two proportions

Call the placebo group 1 and the aspirin group 2. Then p̂₁ = 239 ÷ 11,034 ≈ 0.02166 and p̂₂ = 139 ÷ 11,037 ≈ 0.01259, so the estimated difference is p̂₁ − p̂₂ ≈ 0.00907: about 0.9 percentage points fewer heart attacks with aspirin.

Confidence interval

(p̂₁ − p̂₂) ± z* √( p̂₁(1 − p̂₁)/n₁ + p̂₂(1 − p̂₂)/n₂ )

Here SE = √(0.02166 × 0.97834 ÷ 11,034 + 0.01259 × 0.98741 ÷ 11,037) ≈ 0.00175. A 95% interval is 0.00907 ± 1.96 × 0.00175, or about 0.0056 to 0.0125. Aspirin lowered the five-year heart attack rate by somewhere between about 0.6 and 1.2 percentage points. Zero is not in the interval.

Significance test

H₀: p₁ = p₂ (no difference) against Hₐ: p₁ ≠ p₂. If H₀ is true the two groups share one rate, so the test uses the pooled proportion: p̂ = (239 + 139) ÷ (11,034 + 11,037) = 378 ÷ 22,071 ≈ 0.01713.

z = (p̂₁ − p̂₂) ÷ √( p̂(1 − p̂)(1/n₁ + 1/n₂) )

SE = √(0.01713 × 0.98287 × (1/11,034 + 1/11,037)) ≈ 0.00175, so z ≈ 0.00907 ÷ 0.00175 ≈ 5.19. A z of 5.19 is far in the tail: the two-sided p-value is about 0.0000002. Chance alone almost never splits 378 heart attacks this unevenly between two groups of equal size.

Conditions: two independent random samples or random assignment; at least about 10 successes and 10 failures in each group (139 and 239 both pass); each sample less than 10% of its population if sampling without replacement.

Difference of two means

For quantitative data, compare x̄₁ − x̄₂. Each sample has its own standard deviation, and the standard error combines them:

SE = √( s₁²/n₁ + s₂²/n₂ )    t = (x̄₁ − x̄₂) ÷ SE    interval: (x̄₁ − x̄₂) ± t* × SE

This is the two-sample t procedure, often called Welch's t. Software finds the degrees of freedom with a long formula. By hand, a safe choice is the smaller of n₁ − 1 and n₂ − 1, which gives a slightly wider interval and a slightly larger p-value than software.

Worked example (made-up data). A school tries a new homework app in one randomly chosen group. 25 students using it average 72.4 on a test (s = 8.1); 30 who don't average 67.9 (s = 9.5).

• Difference: 72.4 − 67.9 = 4.5 points.
• SE = √(8.1²/25 + 9.5²/30) = √(2.624 + 3.008) ≈ 2.37.
• t = 4.5 ÷ 2.37 ≈ 1.90 with df = 24 (the smaller of 24 and 29). The two-sided p-value is about 0.07.
• 95% interval: t* = 2.064, so 4.5 ± 2.064 × 2.37 = 4.5 ± 4.9, about −0.4 to 9.4 points.

The interval includes 0, matching p > 0.05: the data can't rule out no effect. It also includes 9 points, a big effect. “Not significant” here means “not enough data to tell”, not “the app does nothing”.

Conditions: independent random samples or random assignment; each group roughly normal, or large enough (about 30 or more) unless strongly skewed or with outliers; the two groups independent of each other.

Randomization tests: shuffle the labels

Formulas aren't the only way. If the treatment truly does nothing, each person's result would have been the same whichever group they landed in, so the group labels are just random tags. A randomization test (or permutation test) asks: if we shuffled the labels again and again, how often would the difference be as large as the one we saw?

1. Compute the real difference between the groups.
2. Pool all the values, shuffle, and deal them back into two groups of the original sizes.
3. Compute the difference for the shuffled groups. Repeat thousands of times.
4. The p-value is the fraction of shuffles at least as extreme as the real difference.

Try it. The first data set is made up: typing speeds (words per minute) of 16 students randomly split into 8 typing with music and 8 in silence.

Randomization tests need no normal curve and few conditions, only that labels could have been swapped under H₀. They are also exactly how random assignment protects an experiment: the p-value comes from the same coin flips that built the groups.

Paired versus independent samples

In 1876 Charles Darwin published an experiment on corn (Zea mays). He grew plants from cross-fertilized and self-fertilized seed in four pots, pairing a crossed plant with a self-fertilized one, and measured their final heights to the nearest eighth of an inch. Fifteen pairs, in four pots:

These are paired data. The two plants in a pair shared a pot, soil and light, so they are not independent, and the natural question is about the differences within pairs. Subtract (cross − self) in each pair and you have a single list of 15 numbers. Then use the one-sample methods from Inference on that list:

• Mean difference d̄ = 2.62 inches, standard deviation s_d = 4.72.
• SE = 4.72 ÷ √15 ≈ 1.22, t = 2.62 ÷ 1.22 ≈ 2.15 with 14 degrees of freedom, two-sided p ≈ 0.0497 (just under 0.05).
• 95% interval: 2.62 ± 2.145 × 1.22, about 0.0 to 5.2 inches.

The randomization version flips signs instead of shuffling groups. If crossing made no difference, each pair's difference was equally likely to come out positive or negative. With 15 pairs there are 2¹⁵ = 32,768 sign patterns. In R. A. Fisher's 1935 book The Design of Experiments, he used Darwin's data (which Darwin had asked Francis Galton to examine) both for a t test on the differences and, later in the same chapter, for this sign-flipping test, one of the first permutation tests. Counting all 32,768 patterns, 1,726 give a mean difference at least as far from 0 as 2.62 inches, a two-sided p-value of about 0.053, very close to the t test. Switch the simulator to Darwin's pairs to watch it.

How to tell which you have

• Paired: each value in one group is matched to one value in the other: the same person before and after, twins, two plants in one pot, left and right hand. The groups are the same size, and the order of the list matters.
• Independent: separate individuals, with nothing linking a value in one group to a particular value in the other. The groups can differ in size.

Analysing paired data as if independent is a real mistake, because it uses the wrong standard error. Usually pairing removes variation between pairs and makes the comparison sharper. In Darwin's data, the pots didn't differ much, so the independent analysis happens to give a smaller p-value (about 0.02), but that number answers a question about 30 unrelated plants that the experiment never asked. Use the method that matches the design.

Practical versus statistical significance

Statistically significant means “hard to explain by chance”. It says nothing about size. With 22,071 people, even a modest effect gives a huge z. So always ask how big the difference is, in units that matter.

The aspirin result can be stated two true ways:

• Relative risk reduction: heart attacks fell from 2.17% to 1.26%, a 44% reduction. This sounds large.
• Absolute risk reduction: 0.9 percentage points over about five years. Put another way, 1 ÷ 0.00907 ≈ 110: about 110 healthy men would need to take aspirin for five years to prevent one heart attack. This is the number needed to treat.

Both are correct; a headline that gives only the relative one is telling half the story (see Misleading Statistics). And benefits must be weighed against harms measured in the same trial: the aspirin group had 23 hemorrhagic strokes against 12 on placebo (P = 0.06, not significant at 5%), and 169 ulcers against 138. Total deaths were nearly equal, 217 against 227. The report also found the heart attack benefit only among men aged 50 or older. Whether a 0.9-point drop is worth it is a medical judgment, not a statistical one. This page is about statistics, not medical advice.

The reverse also happens. A small study, like the homework app above, can find a large difference that isn't significant. Report the confidence interval: it shows the size and the uncertainty together, as the Inference unit advises, rather than a bare p-value.

Choosing a method

Common mistakes

• Adding standard errors. Add the squares (variances), then take the square root.
• Treating paired data as independent, or the reverse. Look at the design, not just the numbers.
• Using the pooled proportion in the interval. Pool only for the test of p₁ = p₂.
• Reading “not significant” as “no difference”. Check the interval: it may include big effects too.
• Reading “significant” as “important”. Give the size of the difference, ideally both absolute and relative.
• Claiming cause from an observational comparison. Only random assignment supports a cause-and-effect conclusion.
• Mixing up the order. Say which group is 1 and which is 2, and keep the sign of the difference consistent.

Practice

History

• 1876 Charles Darwin publishes The Effects of Cross and Self Fertilisation in the Vegetable Kingdom, with his paired corn heights.
• 1935 R. A. Fisher's The Design of Experiments reanalyses Darwin's pairs with a t test and with an early permutation test that flips the signs of the differences.
• 1982 The Physicians' Health Study begins on September 20, randomly assigning 22,071 doctors to aspirin or placebo.
• 1988 After a December 1987 recommendation from its monitoring board, the aspirin part of the trial officially ends early on January 25.
• 1989 The final report on aspirin appears in the New England Journal of Medicine.

Quiz

Ten questions. Pick an answer for each, then check.