Misleading Statistics

Berkeley, 1973

In fall 1973 the graduate school of the University of California, Berkeley, looked at its admissions. Across the whole university about 44% of 8,442 men who applied were admitted, against about 35% of 4,321 women. A gap that size, in that many applicants, is far too big to be chance. Was the university biased against women?

The statisticians Peter Bickel, Eugene Hammel and J. William O'Connell looked department by department, since each department made its own decisions. In the six largest departments the overall gap was even wider, 44.5% for men and 30.4% for women. But inside the departments the picture flipped:

Women did better in four of the six departments, and the two where men did better were close. The gap in the total came from where people applied. Most men applied to A and B, which admitted most applicants. Most women applied to C, D, E and F, which turned most applicants away, men and women alike. The authors' 1975 paper in Science concluded that, once the departments were taken into account, there was “a small but statistically significant bias in favor of women”, and that the real question was why women were steered toward the fields that were hardest to get into.

Simpson's paradox

Simpson's paradox happens when a trend that holds in every group (or most groups) reverses, or disappears, when the groups are combined. It is named after Edward Simpson, who described it in 1951, though Karl Pearson and George Udny Yule had noticed it around 1900.

It needs two things:

• A lurking variable (here, the department) that affects the outcome: some departments admit far more applicants than others.
• Unequal group sizes tied to that variable: the two groups being compared are spread across the departments very differently.

The combined rate is a weighted average of the department rates. Men's average is weighted toward the easy departments, women's toward the hard ones, so the combined rates compare the departments as much as they compare men and women.

Misleading graphs

Truncated axes

A bar's height should be proportional to its value. When the vertical axis starts above 0, a small difference fills the whole chart. That can be fine for a line graph that tracks change, if the axis is clearly labeled, but bar charts compare lengths, so their axis should start at 0.

Edward Tufte measures this with the lie factor: the size of the change shown in the graph divided by the size of the change in the data. Starting the axis at 400 above, the bars grow by 150% (from 12 to 30) for a 4.4% change in the data, a lie factor of about 34. An honest graph has a lie factor near 1.

Area and pictures

When a picture is scaled up to show a bigger number, readers judge it by its area. Double both its width and height and it looks four times as big. A 3-D drawing makes this worse by adding volume.

Dual axes

A chart with two y-axes, one on each side, lets the maker choose both scales freely. By stretching or squeezing either axis, almost any two rising series can be made to cross, track each other, or pull apart. Two lines that seem to “move together” on a dual-axis chart may say more about the axis choices than about any link. Ask for two separate charts, or plot both as percent change from the same starting point.

Other things to check on any graph: are the time intervals even, are the units per person or total, and does the scale change partway through?

Base rates and false positives

A screening test checks people with no symptoms for a condition. Two numbers describe it:

• Sensitivity: the percent of people with the condition who test positive.
• False-positive rate: the percent of people without it who test positive anyway.

The question a patient cares about is different: if I test positive, how likely is it that I have it? That depends heavily on the base rate, the percent of people tested who have the condition. Ignoring it is called base-rate neglect.

The clearest way to work it out is with natural frequencies: imagine a crowd and count people, instead of multiplying probabilities. It is the same conditional probability as Bayes' rule, in a form people find much easier.

Things to try: raise the base rate to 20% (a test given only to people with symptoms) and watch the share of true positives jump. Then lower the false-positive rate instead. Which change helps more?

Researchers Gerd Gigerenzer and Ulrich Hoffrage found that when the same problem was given as natural frequencies instead of percentages, the share of people (including doctors) who reasoned correctly rose sharply.

Cherry-picking

Cherry-picking means choosing only the data that support a claim. It takes many forms:

• Choosing the time window. A stock “up 40% since March” may be down over five years. Start and end dates can make almost any trend.
• Choosing the subgroup. Testing a treatment in twenty subgroups and reporting the one where it worked. With twenty tries at the 5% level, one “significant” result is expected by chance (see hypothesis tests).
• Choosing the measure. Reporting the mean when the median tells a different story, or a total when the rate per person tells a different story.
• Dropping inconvenient data as “outliers” without a reason decided in advance.

The defense is to ask what was not shown: the longer time range, the other groups, the other measures.

Regression to the mean

Any measurement mixes a lasting part (skill, health, ability) with a chance part (luck on the day). An extreme result usually has extreme luck in it, and luck doesn't repeat. So whoever scores extremely high or low the first time tends, on average, to score closer to the mean the next time. That is regression to the mean.

Francis Galton found it in 1886: very tall parents tended to have children who were tall, but less extremely so, and very short parents had children closer to average height.

It fools people when something is done right after an extreme result:

• The students with the lowest scores on a first test get tutoring and improve on the second test. Some of that rise would have happened anyway.
• A player has a record season and then a more ordinary one. People blame pressure or a magazine cover.
• A town installs speed cameras at the sites that had the most crashes last year, and crashes there fall the next year.

The fix is a randomized control group: pick the extreme cases, then randomly give the treatment to only some of them. Both groups regress; only the difference between them shows the effect.

Survivorship bias

Survivorship bias happens when you only see the cases that made it through some filter and treat them as the whole picture.

The famous example comes from World War II. Abraham Wald, a mathematician in the Statistical Research Group at Columbia University, worked on estimating how vulnerable different parts of an aircraft were, using damage recorded on planes that came back from missions. The planes that were shot down could not be inspected. Wald's insight was that the missing planes still carry information: if returning planes rarely show hits in some area, that may be because planes hit there did not return. His conclusions could be used to guide where armor should go.

The story is often retold in a simplified form: officers wanted to armor the places with the most bullet holes, and Wald told them to armor the places with none. Be careful with the details. Wald's actual work was a series of eight technical memoranda full of probability calculations, written in 1943 and later used in Korea and Vietnam as well, and it dealt with aircraft in general rather than only bombers. The well-known picture of a plane covered in red dots is a modern illustration, not a figure from Wald.

Survivorship bias is everywhere:

• Studying only successful companies to find the “secrets of success”, while the failed companies that did the same things are gone.
• “Old buildings were built better.” The badly built ones were torn down long ago.
• Investment funds advertised by past returns, where poorly performing funds were closed and drop out of the averages.

A checklist for any statistic

1. Compared to what? A number alone means little. Look for the base rate, the control group, the earlier year.
2. Who is missing? Who wasn't sampled, didn't answer, dropped out or didn't survive?
3. Is there a lurking variable? Would splitting by age, severity or department change the story?
4. What does the graph's axis do? Where does it start, are there two of them, and is the area honest?
5. Why this window or subgroup? Was it chosen before or after looking at the data?
6. Was something extreme picked? Then expect some regression to the mean.

A related trap is a risk reduction given only in relative terms, like “cuts risk by 44%”. Comparing Two Groups shows how to ask for the absolute number too.

Common mistakes

Practice

History

• 1886 Francis Galton describes “regression towards mediocrity” in the heights of parents and their children.
• 1903 George Udny Yule describes how combining tables can create or hide associations, an early form of what is now called Simpson's paradox.
• 1943 Abraham Wald writes his memoranda on estimating aircraft vulnerability from the damage on planes that survived.
• 1951 Edward Simpson publishes the paradox that now carries his name.
• 1954 Darrell Huff publishes How to Lie with Statistics, the classic guide to misleading graphs and numbers.
• 1975 Bickel, Hammel and O'Connell publish their analysis of Berkeley's 1973 admissions in Science.
• 1983 Edward Tufte defines the lie factor in The Visual Display of Quantitative Information.
• 1995 Gigerenzer and Hoffrage show that natural frequencies make Bayesian reasoning far easier.

Quiz

Ten questions. Pick an answer for each, then check.