Jugami › Statistics and Probability › Statistical Inference
Statistics · Unit 5
Statistical Inference
You only ever see one sample. Inference is how you use it to say something honest about everyone, with a margin of error, a level of confidence and a clear idea of how you could be wrong.
In this unit you'll learn how point estimates and margins of error work, how to build confidence intervals for a proportion and a mean (with z and with t), how sample size controls precision, how hypothesis tests and p-values work, what Type I and Type II errors and power are, and the most common ways these results get misread.
Can she really taste the difference?
In the 1920s, at an agricultural research station in England, a scientist named Muriel Bristol claimed she could tell by taste whether the milk or the tea had been poured into the cup first. The statistician Ronald Fisher later turned the story into a famous experiment design: prepare eight cups, four of each kind, in random order, and ask her to pick out the four milk-first cups.
Suppose she can't tell the difference and is just guessing. There are 8C4 = 70 ways to choose 4 cups out of 8, and only one of them is all correct, so a pure guesser gets all eight right with probability 1/70 ≈ 0.014. If she does that, “just guessing” becomes hard to believe. (A guesser gets at least three of the four milk-first cups right with probability 17/70 ≈ 0.24, which is far too common to be convincing.)
A hypothesis test works the same way: assume “nothing special is going on”, work out how surprising the data would be, and decide. Accounts say she identified all eight cups correctly. This unit builds the tools that make that reasoning work for polls, experiments and measurements.
Recap: sampling distributions
A parameter describes the whole population and is usually unknown. A statistic is calculated from a sample and changes from sample to sample.
| What we want | Parameter | Statistic |
|---|---|---|
| Proportion | p | p̂ (“p-hat”) |
| Mean | μ | x̄ (“x-bar”) |
| Standard deviation | σ | s |
In Unit 4's sampling distributions you saw that if you took many random samples of size n:
- the sample means x̄ center on μ with standard error σ/√n;
- the sample proportions p̂ center on p with standard error √(p(1 − p)/n);
- by the central limit theorem, both are approximately normal when n is large enough.
Inference turns this around. In real life you have one sample and don't know p or μ. Because you know how much a statistic typically wobbles, you can say how far from the truth it is likely to be.
Point estimates and margin of error
A point estimate is a single best guess: p̂ for p, x̄ for μ. It is almost never exactly right, so we add a margin of error (ME) that says how far off it could reasonably be. Every confidence interval in this unit has the same shape:
The critical value comes from the normal curve. About 95% of a normal distribution lies within 1.96 standard deviations of the mean (the “2” in the 68–95–99.7 rule, made exact), so for 95% confidence z* = 1.96.
| Confidence level | 80% | 90% | 95% | 99% |
|---|---|---|---|---|
| z* | 1.282 | 1.645 | 1.960 | 2.576 |
Higher confidence needs a bigger z*, so a wider interval. That is the basic trade-off: you can be more confident or more precise, but with the same data you can't have both.
Confidence interval for a proportion
We don't know p, so the standard error uses p̂ in its place. Check three conditions first:
- Random: a random sample or a randomized experiment.
- Independent (10% rule): when sampling without replacement, n is at most 10% of the population.
- Large counts: n·p̂ ≥ 10 and n·(1 − p̂) ≥ 10, so the normal approximation works.
Worked example: later school start
A random sample of 400 students in a large district (made-up data) finds that 248 support a later start time. Build a 95% confidence interval for the proportion of all students in the district who support it.
- p̂ = 248 ÷ 400 = 0.62.
- Conditions: random sample; 400 is far less than 10% of a large district; 400 × 0.62 = 248 and 400 × 0.38 = 152 are both at least 10.
- SE = √(0.62 × 0.38 ÷ 400) = √0.000589 ≈ 0.0243.
- ME = 1.96 × 0.0243 ≈ 0.048, or about 4.8 percentage points.
- Interval: 0.62 ± 0.048 = (0.572, 0.668).
In context: we are 95% confident that between about 57% and 67% of all students in the district support a later start.
Confidence interval for a mean: z, then t
If you somehow knew the population SD σ, the interval for μ would be x̄ ± z* · σ/√n. In practice σ is almost never known, so you use the sample SD s instead. That adds extra uncertainty, especially in small samples, because s itself wobbles.
William Gosset worked out the fix in 1908: use a t distribution instead of the normal. A t curve looks like the standard normal but has heavier tails. Its exact shape depends on the degrees of freedom, df = n − 1. The more data, the closer t gets to z.
| df | 90% | 95% | 99% |
|---|---|---|---|
| 5 | 2.015 | 2.571 | 4.032 |
| 10 | 1.812 | 2.228 | 3.169 |
| 15 | 1.753 | 2.131 | 2.947 |
| 30 | 1.697 | 2.042 | 2.750 |
| 100 | 1.660 | 1.984 | 2.626 |
| ∞ (z*) | 1.645 | 1.960 | 2.576 |
Conditions: Random, Independent (10% rule) and Normal or large: the population is roughly normal, or n ≥ 30. For small samples, check a dot plot for strong skew or outliers.
Worked example: hours of sleep
A random sample of 16 students (made-up data) sleeps a mean of x̄ = 6.8 hours on school nights, with s = 1.2 hours. Build a 95% confidence interval for the mean of all students.
- SE = s/√n = 1.2 ÷ √16 = 1.2 ÷ 4 = 0.3 hours.
- df = 16 − 1 = 15, so t* = 2.131 (from the table).
- ME = 2.131 × 0.3 ≈ 0.64 hours.
- Interval: 6.8 ± 0.64 = (6.16, 7.44) hours.
Using z* = 1.96 by mistake would give ±0.59 hours: an interval that looks more precise than the data can support.
What “95% confident” really means
The 95% describes the method, not one particular interval. If you took many random samples and built a 95% interval from each one, about 95% of those intervals would capture the true parameter. Any single interval either contains it or it doesn't; you just don't know which.
Say: “We are 95% confident that the interval from 0.572 to 0.668 captures the true proportion.” Mean: “95% of intervals built this way capture the truth.”
Try it: raise the confidence level and the bars get longer while the misses get rarer. Raise n and the bars get shorter, but the capture rate stays near the confidence level.
Sample size and margin of error
Because n sits under a square root, the margin of error shrinks slowly: to cut the margin of error in half, you need four times as many people.
To plan a survey for a proportion, solve ME = z*·√(p̂(1 − p̂)/n) for n. If you have no earlier estimate, use p̂ = 0.5, which gives the largest (safest) answer. Always round up.
- ±3 percentage points at 95%: (1.96/0.03)² × 0.25 = 1,067.1, so n = 1,068. That is why many national polls survey about 1,000 people.
- ±5 points: 384.16, so n = 385. ±2 points: exactly 2,401.
- For a mean, n = (z*·σ/E)². With σ = 15 and E = 3 at 95%: (1.96 × 15 ÷ 3)² = 96.04, so n = 97.
The margin of error only covers random sampling error. It says nothing about bias from a bad sample, non-response or loaded questions (see Unit 4 on sampling).
Hypothesis tests: the logic
A confidence interval asks “what values are plausible?” A hypothesis test asks “is this particular claim believable, given the data?”
- Null hypothesis H₀: the “nothing new” claim, always with an equals sign, about a parameter. Example: H₀: p = 0.70.
- Alternative hypothesis Hₐ: what you're looking for evidence of: p < 0.70, p > 0.70 (one-sided) or p ≠ 0.70 (two-sided). Choose it before seeing the data.
- Test statistic: how many standard errors the statistic is from the null value.
- P-value: assuming H₀ is true, the probability of getting a statistic at least as extreme as the one observed (in the direction of Hₐ).
- Significance level α: the cutoff, chosen in advance (0.05 is common). If p-value ≤ α, reject H₀: the result is “statistically significant”. If p-value > α, fail to reject H₀.
A small p-value means the data would be surprising if H₀ were true. It is evidence against H₀, the same way the tea taster's eight-for-eight was evidence against “just guessing”.
Four steps: State H₀, Hₐ and α. Plan: name the test and check the conditions. Do: compute the test statistic and p-value. Conclude: reject or fail to reject, in the context of the problem.
P-values, pictured
When H₀ is true, a z test statistic follows the standard normal curve. The p-value is the area in the tail (or tails) beyond your statistic. For a two-sided Hₐ, count both tails: double the one-tail area.
Worked example: one-proportion z-test
A cereal company claims that 70% of teens eat breakfast on school days. A random sample of 200 teens (made-up data) finds 126 who do. At α = 0.05, is there convincing evidence that the true proportion is less than 70%?
- State. H₀: p = 0.70, Hₐ: p < 0.70, where p is the proportion of all teens who eat breakfast on school days.
- Plan. One-proportion z-test. Random sample; 200 is less than 10% of all teens; large counts using the null value: 200 × 0.70 = 140 and 200 × 0.30 = 60, both ≥ 10.
- Do. p̂ = 126 ÷ 200 = 0.63. SE = √(0.70 × 0.30 ÷ 200) ≈ 0.0324. z = (0.63 − 0.70) ÷ 0.0324 ≈ −2.16. P-value = P(Z ≤ −2.16) ≈ 0.015.
- Conclude. 0.015 < 0.05, so reject H₀. There is convincing evidence that fewer than 70% of teens eat breakfast on school days.
Notice the SE uses p₀ = 0.70, not p̂. In a test you assume H₀ is true, so you use its value. In a confidence interval there is no null value, so you use p̂.
Worked example: one-mean t-test
Bags of chips are labelled 28 g. An inspector weighs a random sample of 25 bags (made-up data): x̄ = 27.6 g, s = 1.0 g. At α = 0.05, is the true mean different from 28 g?
- State. H₀: μ = 28, Hₐ: μ ≠ 28, where μ is the mean weight of all bags.
- Plan. One-sample t-test. Random sample; 25 bags is far less than 10% of production; the weights in the sample show no strong skew or outliers.
- Do. SE = 1.0 ÷ √25 = 0.2. t = (27.6 − 28) ÷ 0.2 = −2.00 with df = 24. Two-sided p-value = 2 × P(t₂₄ ≤ −2.00) ≈ 2 × 0.0285 = 0.057.
- Conclude. 0.057 > 0.05, so fail to reject H₀. We don't have convincing evidence that the mean weight differs from 28 g.
The matching 95% interval, 27.6 ± 2.064 × 0.2 = (27.19, 28.01) g, contains 28: the two methods agree. And “fail to reject” is not the same as “the bags are fine”: a bigger sample might still find a real shortfall.
A one-sided Hₐ: μ < 28 would have given p ≈ 0.028. That's why Hₐ must be chosen before looking at the data. Switching after you see x̄ is cheating.
Type I and Type II errors, and power
Every test decision can be wrong in one of two ways:
| Decision | H₀ is actually true | H₀ is actually false |
|---|---|---|
| Reject H₀ | Type I error (false alarm); probability α | Correct; probability = power |
| Fail to reject H₀ | Correct | Type II error (a miss); probability β |
Power = 1 − β is the chance that the test detects an effect that really exists. Power goes up when:
- the sample size n is larger (the curves get narrower);
- the true effect is bigger (the curves are farther apart);
- α is larger (but then Type I errors become more likely);
- the data are less variable.
Statistical vs practical significance
“Statistically significant” only means “hard to explain by chance alone”. It does not mean large, important or useful.
A huge sample, a tiny effect
A tutoring app is tried by 40,000 students (made-up data). Their mean test score is 0.3 points higher than the known average, on a test where the SD is 10 points. SE = 10 ÷ √40,000 = 0.05, so z = 0.3 ÷ 0.05 = 6 and the p-value is about 0.000000002. Extremely significant, yet 0.3 points is too small to matter to anyone.
The reverse also happens: a small study can miss a large, real effect because its power is low. That's why good reports give a confidence interval for the size of the effect, not just a p-value.
Next: Comparing Two Groups extends these intervals and tests to differences between two groups, with a real aspirin trial where both kinds of significance matter.
Common misreadings
“There is a 95% chance that μ is in this interval.” Once the interval is computed, μ is either in it or not. The 95% is the long-run success rate of the method.
“p = 0.03 means a 3% chance H₀ is true.” The p-value assumes H₀ is true and asks how surprising the data are. It is P(data this extreme | H₀), not P(H₀ | data).
“We failed to reject H₀, so H₀ is true.” Failing to reject means the evidence wasn't strong enough. Absence of evidence is not evidence of absence, especially with a small sample.
“It's significant, so it's important.” See the tutoring app above. Look at the size of the effect.
P-hacking and multiple comparisons
At α = 0.05, each test of a true H₀ has a 5% chance of a false alarm. Run 20 independent tests on pure noise and the chance that at least one comes out “significant” is 1 − 0.95²⁰ ≈ 0.64. Trying many tests, subgroups or outcome measures and reporting only the one that “worked” is called p-hacking or data dredging.
Defences: decide the hypotheses and the analysis before collecting data (pre-registration), report every test you ran, and when you must run many, use a stricter cutoff. The simple Bonferroni rule divides α by the number of tests: 0.05 ÷ 20 = 0.0025, which brings the chance of any false alarm in 20 tests back below 5% (about 4.9%).
Correlation is not causation
A significant test says an association is unlikely to be pure chance. It does not say why. In observational data a hidden third variable (a confounder) can drive both things. Only a well-designed randomized experiment supports a cause-and-effect conclusion. You'll see more in Unit 6: correlation and causation.
Common mistakes
Writing hypotheses about the statistic. “H₀: p̂ = 0.70” is wrong. Hypotheses are about the parameter: H₀: p = 0.70.
Using z* for a mean when σ is unknown. With s, use t* with df = n − 1. With small n the difference is large.
Mixing up p̂ and p₀ in the standard error. Intervals use p̂; tests use the null value p₀.
Choosing a one-sided Hₐ after seeing the data. It halves the p-value unfairly. Choose before you look.
Confusing SE with margin of error. ME = critical value × SE. At 95% it is about twice the SE.
Skipping the conditions or the context. “Reject H₀” alone is not a conclusion. Say what it means for the real question.
Practice
People behind the ideas
1908 William Sealy Gosset, a chemist and brewer at Guinness in Dublin, publishes “The Probable Error of a Mean” in Biometrika under the pen name “Student”, because Guinness had its staff publish under pseudonyms. His work on small samples gives us the t distribution, later named Student's t.
1925 Ronald Fisher's book Statistical Methods for Research Workers popularizes “tests of significance” and suggests a probability of one in twenty (0.05) as a convenient cutoff. Fisher did not intend 0.05 to be a fixed rule.
1933 Jerzy Neyman and Egon Pearson treat a test as a decision between two hypotheses, with the significance level α set in advance and with attention to both kinds of error: rejecting a true hypothesis and missing a false one.
1935 Fisher's The Design of Experiments describes the lady tasting tea experiment, inspired by Muriel Bristol, and argues that a null hypothesis is never proved, only possibly disproved.
1937 Neyman publishes “Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability”, introducing confidence intervals and confidence levels.
2016 The American Statistical Association releases its Statement on Statistical Significance and P-Values (Ronald Wasserstein and Nicole Lazar), with six principles. Two of them: p-values “do not measure the probability that the studied hypothesis is true”, and a p-value “does not measure the size of an effect or the importance of a result.”
Quiz
Ten questions. Pick an answer for each, then check.
Talk about it
- A news story says a poll found 52% support “with a margin of error of ±3 points”. Is the race “too close to call”? What does the margin of error leave out?
- Why might a medical researcher choose α = 0.01 instead of 0.05? When could a Type II error be the worse mistake?
- Some scientists have argued that the phrase “statistically significant” should be dropped entirely. What would be lost, and what would be gained?
- A company tests 20 ad designs and announces that one “significantly” increased sales. What questions would you ask before believing it?
- Think of a claim from sports, health or school that you could test with a small study. What would H₀ and Hₐ be, and what could go wrong?
Sources
- Wikipedia, Lady tasting tea: Fisher's eight-cup design in The Design of Experiments (1935), inspired by Muriel Bristol; 70 possible selections; the null hypothesis “never proved or established, but is possibly disproved”; accounts that she identified all eight cups.
- Wikipedia, William Sealy Gosset: “The Probable Error of a Mean”, Biometrika (1908), published as “Student” while working for Guinness in Dublin.
- Wikipedia, Statistical significance: history, citing Fisher (1925), Quinn and Keough (2002), Neyman and Pearson (1933) and Neyman (1937): the one-in-twenty cutoff, α set in advance, and confidence intervals introduced by Neyman in 1937.
- American Statistical Association, press release, March 7, 2016, listing the six principles; Ronald L. Wasserstein and Nicole A. Lazar, “The ASA Statement on p-Values: Context, Process, and Purpose”, The American Statistician 70(2), 129–133 (2016), doi:10.1080/00031305.2016.1154108.
- All critical values, intervals, p-values and power figures on this page were computed exactly (normal and t distributions) and rounded.
- All data sets on this page (the school-start survey, sleep hours, phone battery lives, breakfast survey, chip bags, the power example's test scores, the tutoring app and the p-hacking “treatments”) are made up for teaching.