Jugami › Statistics and Probability › Chi-Square Tests
Statistics · Unit 8
Chi-Square Tests
How to tell whether counts in categories fit a model, and whether two categorical variables are related, using one number: χ².
Some data come as numbers, like heights or test scores. Much data comes as categories: which face a die shows, which lunch a student picks, which party a voter supports. With categories, all you can do is count. This unit shows how to compare the counts you observed with the counts a model says you should expect, and how to decide whether the gap is bigger than chance would make. It uses the hypothesis tests and p-values from Unit 5 and the two-way tables from Unit 2.
Weldon's dice, 1900
In 1894 the zoologist W. F. Raphael Weldon set out to test whether his dice were fair. He and his helpers threw 12 dice at a time, 26,306 times, and each time counted how many dice showed a 5 or a 6. If the dice were fair, a 5 or 6 has probability 1/3 on each die, so the counts should follow a binomial distribution with n = 12 and p = 1/3. Weldon sent his table to Francis Galton in a letter dated 2 February 1894.
The counts looked close to the binomial, but not exactly. Some differences were positive, some negative. Was the gap just chance, or were the dice off? Nobody had a good way to judge a whole table of differences at once.
In 1900 Karl Pearson published one. In the Philosophical Magazine he combined all the differences into a single number, which he called χ² (chi-square, said “kye-square”), and worked out how big it tends to be by chance alone. Weldon's dice were his first example. Pearson's answer: a gap as large as Weldon's would turn up by chance only about once in 62,500 tries. The dice favored 5s and 6s.
| Dice showing 5 or 6 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10+ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Observed | 185 | 1,149 | 3,265 | 5,475 | 6,114 | 5,194 | 3,067 | 1,331 | 403 | 105 | 18 |
| Expected (fair) | 203 | 1,216 | 3,345 | 5,576 | 6,273 | 5,018 | 2,927 | 1,255 | 392 | 87 | 14 |
Look at the pattern: too few throws with 0–4 fives and sixes, too many with 5 or more. Done the modern way, with the last three columns grouped so every expected count is at least 5, the data give χ² ≈ 35.5 on 10 degrees of freedom, a p-value of about 0.0001. Pearson blamed the pips: on ordinary dice they are hollowed out, so the six face is a little lighter than the one face opposite it. (A modern repeat of the experiment, below in the history, suggests the real cause may be dice that are not perfect cubes.)
Pearson's χ² is still one of the most used tests in science. By the end of this unit you'll be able to run it yourself.
Categorical data and counts
A categorical variable puts each individual into a group: eye color, favorite sport, yes or no. You can't average categories, but you can count them. Those counts are the data for every test in this unit.
- One categorical variable: a one-way table of counts, like how many times each face of a die came up. The question is whether the counts fit a claimed distribution.
- Two categorical variables: a two-way table (or contingency table), like grade level by favorite way to study. The question is whether the variables are related.
Counts, not percents. The χ² formula needs counts. If you're given percents, turn them back into counts first. 30% of 50 people and 30% of 5,000 people carry very different amounts of evidence.
Observed and expected counts
The observed counts (O) are what you actually saw. The expected counts (E) are what the null hypothesis says you'd see on average:
Roll a fair die 60 times and you expect 60 × 1/6 = 10 of each face. You won't get exactly 10 of each; chance always moves the counts around. The question a χ² test answers is whether the observed counts are too far from the expected ones to blame on chance.
Expected counts don't have to be whole numbers. If 1/3 of 50 throws should show a 5 or 6, the expected count is 16.67. That's an average over many repeats, not a prediction for one. Don't round it.
The chi-square statistic
For each category, find how far the observed count is from the expected count, square it, and divide by the expected count. Then add up over all the categories:
- Squaring makes every gap count, too many or too few, and stops positives and negatives from canceling.
- Dividing by E scales each gap. Being 5 off when you expected 10 is a big miss; being 5 off when you expected 1,000 is tiny.
- χ² is never negative. It is 0 only when every observed count equals its expected count. Bigger χ² means a worse fit and stronger evidence against the null hypothesis.
Each term (O − E)² ÷ E is that category's contribution. The largest contributions show you where the data and the model disagree most.
Move the counts, watch χ²
Drag the top of any bar, or pick a category and use the buttons. The dashed lines are the expected counts. Watch how each category's contribution, the total χ² and the p-value change.
Things to try:
- Press Simulate a sample if H₀ is true a dozen times. Even a perfectly fair die rarely gives exactly 10 of each, and χ² bounces around. Only about 1 run in 20 should land in the 5% tail.
- Pull one category far from its dashed line. Its contribution grows with the square of the gap: twice the gap, four times the contribution.
- In the candy data, the same miss of 6 counts adds much more for a rare color (small E) than for a common one.
Degrees of freedom and p-values
If the null hypothesis is true, χ² follows (approximately) a chi-square distribution. There is a different curve for each number of degrees of freedom (df). All of them start at 0 and are skewed right; the mean of each curve equals its df.
For a goodness-of-fit test, df = (number of categories) − 1. Why minus 1? The counts must add up to n, so once you know all but one of them, the last is fixed. Only k − 1 counts are free to vary.
The p-value is the area under the curve to the right of your χ²: the chance of a fit this bad or worse if H₀ is true. It is always one-sided, because any kind of misfit makes χ² larger.
| df | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| χ² critical value | 3.84 | 5.99 | 7.81 | 9.49 | 11.07 | 12.59 |
As a rough guide, a χ² near its df is unremarkable. A χ² far above its df is suspicious.
The goodness-of-fit test
A goodness-of-fit test asks whether one categorical variable follows a claimed distribution.
Worked example: the cafeteria claim
A school cafeteria says students choose pizza 40% of the time, the salad bar 30%, sandwiches 20% and soup 10%. On a random day, 200 students chose: pizza 92, salad 50, sandwiches 44, soup 14. Do the data fit the claim?
- Hypotheses. H₀: the proportions are 0.40, 0.30, 0.20, 0.10. Hₐ: at least one proportion is different.
- Expected counts. 200 × 0.40 = 80, 200 × 0.30 = 60, 200 × 0.20 = 40, 200 × 0.10 = 20. All are at least 5.
- Statistic. χ² = (92 − 80)²/80 + (50 − 60)²/60 + (44 − 40)²/40 + (14 − 20)²/20 = 1.80 + 1.67 + 0.40 + 1.80 = 5.67.
- df and p-value. 4 categories, so df = 3. The p-value is P(χ² ≥ 5.67) ≈ 0.13. (5.67 is below the critical value 7.81.)
- Conclusion. At α = 0.05, fail to reject H₀. These data are consistent with the cafeteria's claim. They don't prove it is right, only that the differences are the size chance often produces.
“We accept H₀, so the claim is true.” A χ² test can show a model fits badly; it can't prove a model is right. Say “fail to reject” and “consistent with”.
Two-way tables and expected counts
Now two variables. A survey of 200 randomly chosen students at a high school recorded each student's grade and how they prefer to study:
| Grade | Alone | With friends | Online group | Total |
|---|---|---|---|---|
| 9th | 30 | 50 | 20 | 100 |
| 12th | 45 | 35 | 20 | 100 |
| Total | 75 | 85 | 40 | 200 |
If grade and study style were independent, both grades would split the same way as the whole school: 75 of 200 study alone, so 75/200 of the 100 ninth graders, 37.5, would be expected to. In general:
| Grade | Alone | With friends | Online group |
|---|---|---|---|
| 9th | 100 × 75 ÷ 200 = 37.5 | 100 × 85 ÷ 200 = 42.5 | 100 × 40 ÷ 200 = 20 |
| 12th | 37.5 | 42.5 | 20 |
The expected table has the same row and column totals as the observed one. That's a good check on your arithmetic.
The test of independence
A chi-square test of independence uses one random sample, classified by two variables, and asks whether the variables are associated.
Worked example: grade and study style
- Hypotheses. H₀: grade and study style are independent (no association). Hₐ: they are associated.
- Statistic. Add (O − E)²/E over all six cells: 1.50 + 1.32 + 0 + 1.50 + 1.32 + 0 = 5.65.
- Degrees of freedom. For a table with r rows and c columns, df = (r − 1)(c − 1). Here (2 − 1)(3 − 1) = 2.
- p-value. P(χ² ≥ 5.65) with df = 2 is about 0.059. Just above 0.05: 5.65 falls short of the critical value 5.99.
- Conclusion. At α = 0.05, fail to reject H₀. There is some evidence that ninth and twelfth graders study differently (ninth graders lean toward studying with friends), but not quite enough at the 5% level.
Why (r − 1)(c − 1)? Once the row and column totals are fixed, you can fill in only r − 1 rows and c − 1 columns freely; the rest is forced by the totals. R. A. Fisher pointed this out in 1922, correcting the count Pearson had used for tables.
Independence or homogeneity?
The same arithmetic answers a slightly different question. A test of homogeneity takes separate random samples from two or more populations (say, 100 ninth graders and 100 twelfth graders, sampled separately) and asks whether the distribution of one variable is the same in each. The χ², df and p-value are found exactly the same way. What differs is the design and the wording of the hypotheses:
| Test | Data | Question | df |
|---|---|---|---|
| Goodness of fit | One sample, one variable | Do the counts fit a claimed distribution? | k − 1 |
| Independence | One sample, two variables | Are the two variables associated? | (r − 1)(c − 1) |
| Homogeneity | Several samples, one variable | Is the distribution the same in each population? | (r − 1)(c − 1) |
Association isn't causation. A significant test of independence says the variables are related in the population. Unless the groups came from random assignment, it doesn't say one causes the other.
Conditions
The χ² curve is only an approximation to how the statistic really behaves. It works well when:
- Random: the data come from a random sample (or samples), or a randomized experiment.
- Independent: individual observations are independent. When sampling without replacement, the sample is less than 10% of the population.
- Large counts: every expected count is at least 5. (Some books allow up to 20% of expected counts between 1 and 5.) Check the expected counts, not the observed ones.
- Counts: the table holds counts of individuals, each counted in exactly one cell.
If an expected count is too small, you can combine neighboring categories, as was done with Weldon's 10, 11 and 12, or collect more data.
Checking the observed counts for “at least 5”. An observed count of 2 is fine if its expected count is 8. The condition is about what the model expects.
Common mistakes
Using percents or proportions instead of counts. The formula only works on counts.
Dividing by O instead of E. It's (O − E)² ÷ E, the expected count.
Getting the degrees of freedom wrong. Goodness of fit: categories − 1, not n − 1. Two-way table: (rows − 1)(columns − 1), not counting the total row and column.
Doubling the p-value. A χ² test is always right-tailed. There's no “two-sided” version.
Saying the data prove H₀. A large p-value means the data are consistent with H₀, not that H₀ is true.
Rounding expected counts to whole numbers. Keep the decimals until the end.
Reading cause into a test of independence. Association in an observational study doesn't show cause and effect. Splitting a two-way table by a third variable can even reverse an association: see Simpson's paradox in Misleading Statistics.
Practice
History
- 1894 W. F. R. Weldon reports 26,306 throws of 12 dice to Francis Galton, counting the 5s and 6s each time.
- 1900 Karl Pearson publishes the χ² test of goodness of fit in the Philosophical Magazine, using Weldon's dice as his first illustration.
- 1922 R. A. Fisher shows that for a two-way table the degrees of freedom are (r − 1)(c − 1), not the number of cells, which changes many of the p-values computed with Pearson's method.
- 2009 Zacariah Labby builds a machine that throws 12 ordinary dice and photographs them, and repeats Weldon's 26,306 throws. Counting 5s and 6s together, his dice fit the fair model. But a χ² test on all six faces gives χ² = 25.0 (p = 0.00014): these dice were biased too, toward 1 and 6. Measuring them, he found the 1–6 axis slightly shorter than the others, so the 1 and 6 faces were a little larger.
Quiz
Ten questions. Pick an answer for each, then check.
Talk about it
- Weldon's dice gave a tiny p-value, yet the counts looked close to the fair model. How can a small difference be “significant”? Would the same small difference be significant with 260 throws?
- If someone's data fit a model too well, with χ² close to 0 again and again, should that make you suspicious? Why?
- Why does it matter that the squared gaps are divided by the expected counts rather than just added up?
- A survey finds that favorite subject and gender are associated. What can and can't you conclude?
- Design a study that would call for a test of homogeneity rather than a test of independence.
Sources
- Karl Pearson, “On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling”, Philosophical Magazine, Series 5, 50(302), 157–175 (1900), doi:10.1080/14786440009463897.
- R. L. Plackett, “Karl Pearson and the Chi-Squared Test”, International Statistical Review 51(1), 59–72 (1983), doi:10.2307/1402731: Weldon's 26,306 throws of 12 dice as Pearson's first two illustrations.
- Zacariah Labby, “Weldon's Dice, Automated”, Chance 22(4), 6–13 (2009), doi:10.1080/09332480.2009.10722977: Weldon's letter to Galton of 2 February 1894, Pearson's P = 0.000016 (“62,499 to 1”), Pearson's hollowed-pip explanation, and the automated repeat (χ² = 25.0, p = 0.00014 for the six faces; the shorter 1–6 axis).
- Weldon's counts as published in the WeldonDice data set (R package vcd), with 10 or more 5s and 6s grouped; the grouped test (χ² ≈ 35.5, df 10, p ≈ 0.0001) follows Stephen Stigler's Statistics 244 notes, chapter 7. Expected counts and χ² recomputed for this page.
- R. A. Fisher, “On the Interpretation of χ² from Contingency Tables, and the Calculation of P”, Journal of the Royal Statistical Society 85(1), 87–94 (1922), JSTOR 2340521.
- Wikipedia, Pearson's chi-squared test: degrees of freedom, expected counts for contingency tables, and the expected-count condition.
- The cafeteria, study-style, candy and die data on this page are made up for teaching. Critical values and p-values were computed for this page.