Jugami › Statistics and Probability › Correlation and Regression
Statistics · Unit 6
Correlation and Regression
When two things rise and fall together, a scatterplot shows it, r measures it and a least-squares line predicts it. None of them can tell you why.
Countries that eat more chocolate win more Nobel Prizes per person, and the link is strong. Does chocolate make people smarter? To answer that, you need the tools in this unit: a scatterplot to see the pattern, the correlation r to measure it, a least-squares line to predict from it, residuals and r² to check the fit, an eye for single points that pull the line around, and a clear sense of when a correlation can, and can't, show cause and effect.
Does chocolate win Nobel Prizes?
In October 2012 the New England Journal of Medicine, one of the world's best-known medical journals, printed a short note by Franz Messerli. For 23 countries he plotted how much chocolate people eat per person each year against how many Nobel laureates the country has per 10 million people. The points rose together strongly: the correlation was r = 0.791, with a p-value below 0.0001. Leaving out one country, Sweden, pushed it up to 0.862.
Messerli even worked out a slope from his line: it would take about 0.4 kg more chocolate per person per year to raise a country's number of Nobel laureates by one.
So should a country hand out chocolate to win prizes? Keep your answer in mind. By the end of this unit you'll be able to say exactly what this graph shows, what it doesn't, and what kind of study could settle the question.
Reading a scatterplot
A scatterplot shows two quantitative variables measured on the same individuals: each individual is one dot. Put the explanatory variable (the one you think may explain or predict the other) on the horizontal x-axis, and the response variable on the vertical y-axis. In the chocolate graph each dot is a country, chocolate is x and Nobel laureates are y.
To describe a scatterplot, say four things, always in context:
- Direction. Positive if y tends to go up as x goes up; negative if y tends to go down.
- Form. Do the points follow a straight line (linear), a curve, or separate clusters?
- Strength. How closely do the points follow that form? Tightly packed is strong; loosely scattered is weak.
- Unusual features. Outliers that sit away from the pattern, gaps or clusters.
Example. “There is a strong, positive, linear association between arm span and height for these students, with no clear outliers. Students with longer arm spans tend to be taller.”
The correlation coefficient r
The correlation coefficient r is a single number that measures the direction and strength of the linear association between two quantitative variables. To find it, turn every x and every y into a z-score, multiply each pair, add up the products and divide by n − 1:
A point that is above average in both x and y (or below average in both) gives a positive product. A point that is above in one and below in the other gives a negative product. So r is positive when most points sit in the “both above” and “both below” corners.
Worked example. Five students (made-up data) record hours studied, x, and quiz score out of 10, y.
| x | y | zx | zy | zx · zy |
|---|---|---|---|---|
| 1 | 2 | −1.265 | −1.265 | 1.6 |
| 2 | 3 | −0.632 | −0.632 | 0.4 |
| 3 | 5 | 0 | 0.632 | 0 |
| 4 | 4 | 0.632 | 0 | 0 |
| 5 | 6 | 1.265 | 1.265 | 1.6 |
The products add up to 3.6, so r = 3.6 ÷ (5 − 1) = 0.9: a strong positive linear association. In practice a calculator does this, but now you know what it is adding up.
What r does and doesn't do
- r is always between −1 and 1. It is exactly 1 or −1 only when every point lies on one straight line.
- The sign gives the direction. The closer r is to −1 or 1, the more tightly the points hug a line; r near 0 means no linear pattern.
- r has no units. Changing units (inches to centimetres, °F to °C) does not change it.
- Swapping x and y does not change r.
- r measures only linear association. A strong curve can have r near 0.
- r is not resistant: a single outlier can push it a long way up or down.
Guess the correlation
Getting a feel for r takes practice. A cloud has to be quite tight before r reaches 0.9, and a loose cloud with r = 0.5 can look like almost nothing. Play a few rounds to calibrate your eye.
Same r, different stories: Anscombe's quartet
In 1973 the statistician Francis Anscombe published four small data sets to make one point: always graph your data. All four have the same mean of x (9), the same mean of y (7.50), the same correlation (r = 0.816) and the same least-squares line (ŷ = 3.00 + 0.500x). They look nothing alike.
- Set I is what r = 0.816 “should” look like: a linear cloud.
- Set II follows a smooth curve perfectly. A line is the wrong model.
- Set III lies almost exactly on a line except for one point. Without that outlier, r would be almost exactly 1.
- Set IV has ten points at the same x. One far-out point creates the whole “association”; without it, r cannot even be computed.
Numbers summarize; they don't show. Before you trust r or a regression line, look at the scatterplot.
The least-squares regression line
A regression line describes how the response y changes as x changes, and lets you predict y from x. We write it as
where ŷ (“y-hat”) is the predicted y, b is the slope and a is the y-intercept. For each data point, the residual is the gap between what happened and what the line predicted:
Many lines could go through a cloud of points. The least-squares regression line (LSRL) is the one that makes the sum of the squared residuals as small as possible. Squaring makes every residual count as positive and punishes big misses much more than small ones.
Finding the line from summary statistics
You don't have to search for the line. From the means, the standard deviations and r:
The second formula says the LSRL always passes through the point (x̄, ȳ). The first says the slope has the same sign as r.
Worked example. For a class of students (made-up data), arm span x has mean x̄ = 165 cm and SD sx = 9 cm; height y has mean ȳ = 166 cm and SD sy = 8 cm; and r = 0.9.
- Slope. b = 0.9 × 8 ⁄ 9 = 0.8
- Intercept. a = 166 − 0.8 × 165 = 166 − 132 = 34
- Line. predicted height = 34 + 0.8 × (arm span), both in cm.
- Predict. For an arm span of 170 cm: ŷ = 34 + 0.8 × 170 = 170 cm.
- Residual. A student with that arm span who is actually 173 cm tall has residual 173 − 170 = +3 cm: the line under-predicted by 3 cm, and the point lies above the line.
Why “regression”? Put the slope formula in z-scores and it reads ẑy = r · zx. Because r is between −1 and 1, a predicted y is always fewer SDs from its mean than x is from its mean. Francis Galton saw this in 1886 with parents' and children's heights: children of very tall parents were tall, but on average less extreme than their parents. He called it regression toward the mean, and the name stuck to the whole method.
Interpreting the line, and extrapolation
- Slope: “For each additional 1 cm of arm span, the predicted height increases by 0.8 cm.” Say “predicted” (or “on average”): individual students vary around the line.
- Intercept: “When the arm span is 0 cm, the predicted height is 34 cm.” That is meaningless here, since nobody has an arm span of 0. The intercept is only worth interpreting when x = 0 makes sense and is near the data.
Extrapolation
Extrapolation means using the line to predict far outside the range of x-values in the data. The line only describes the data you have; nothing guarantees that the pattern continues.
For children aged 2 to 10, a made-up growth line might be predicted height = 80 + 6 × (age), in cm. It works well at age 8 (128 cm). At age 40 it predicts 80 + 240 = 320 cm, more than 3 m tall. People stop growing; the line doesn't know that.
Residuals and residual plots
A residual plot graphs each residual (y − ŷ) against x, with a horizontal line at 0. For the least-squares line, the residuals always add up to 0, so the plot is centred on that line. It magnifies anything the line is missing.
- Random scatter, no pattern: a line is a reasonable model.
- A curved pattern: the relationship is not linear; a line is the wrong model.
- A fan shape (residuals spreading out as x grows): predictions are less precise for larger x.
r²: how much does the line explain?
The coefficient of determination, r², is the fraction of the variation in y that is explained by the least-squares line on x. It compares the squared residuals around the line with the squared deviations around the plain mean ȳ:
For the five study-hours students above, the squared deviations from ȳ = 4 add up to 10, and the squared residuals from the line ŷ = 1.3 + 0.9x add up to 1.9. So r² = 1 − 1.9 ⁄ 10 = 0.81, which is 0.9² as it should be.
Say it in context: “About 81% of the variation in quiz scores is explained by the linear relationship with hours studied.” The other 19% is left over in the residuals: other influences, and chance.
r² loses the sign: r = 0.9 and r = −0.9 both give r² = 0.81. And a high r² is not proof that a line is the right model. Anscombe's sets I to IV all have r² ≈ 0.67.
Outliers and influential points
- An outlier in regression is a point with a large residual: far above or below the line.
- A point has high leverage if its x-value is far from x̄. Like a long lever, it can tilt the line a lot.
- A point is influential if removing it would change the slope, intercept or r a lot. High-leverage points are often influential, especially when they are off the trend.
Notice the middle outlier: it has a huge residual and hurts r badly, yet the slope barely moves, because its x is right at x̄. Far-right points have the most leverage. Whether they are influential depends on whether they agree with the rest.
Don't just delete points you don't like. Check whether an unusual point is a mistake; if it is real, report results with and without it.
Correlation is not causation
A strong correlation says two variables move together. It does not say that one causes the other. There are several other explanations to rule out first:
- A lurking variable: a variable that is not among those studied but affects both of them. A classic made-up example: towns sell more ice cream on days when more people drown. Hot weather drives both.
- Reverse direction: y might cause x, rather than x causing y.
- Coincidence: with enough variables to compare, some will line up by chance (see p-hacking in Unit 5).
Two variables are confounded when their effects on the response can't be told apart. If richer countries eat more chocolate and spend more on universities and research, then chocolate and wealth are confounded, and the chocolate graph can't separate them.
Messerli's data are also country-level averages. Even a real link between countries would not show that the individual people who eat more chocolate are the ones who win prizes.
A real case: hormone therapy and heart disease
By the 1990s, many observational studies had found that women who took hormone therapy after menopause had less heart disease than women who did not. In 1996 the US Preventive Services Task Force recommended (grade “B”) that doctors counsel women about hormone therapy's possible benefits, which observational studies suggested included less heart disease. But the women in those studies had chosen hormone therapy, and observational data can't separate the effect of the hormones from the other ways those women differed from women who didn't take them.
The Women's Health Initiative tested it with a randomized experiment. In one trial, 16,608 women aged 50 to 79 were randomly assigned to take estrogen plus progestin or a placebo. The trial was stopped early, after an average of 5.2 years, and the results were published in July 2002. Hormone therapy had not protected the heart: the hazard ratio for coronary heart disease was 1.29 (95% CI 1.02 to 1.63), and there was more breast cancer and stroke as well, though fewer hip fractures. In 2002 the Task Force gave a “D” grade, a recommendation against using hormone therapy to prevent chronic conditions, and it reaffirmed that in 2017 and 2022.
How to get evidence of cause and effect: a well-designed randomized experiment. Random assignment makes the groups alike on average in every lurking variable, measured or not, so a clear difference in the response can be put down to the treatment (Unit 7 shows how to design one). When an experiment is impossible or unethical (no one may assign people to smoke), causal claims need many studies that agree, a plausible mechanism and a dose-response pattern.
Back to chocolate: the graph shows a strong country-level association, nothing more. To test whether chocolate helps, you would have to randomly assign people to eat it, and Nobel Prizes would be a slow outcome to wait for.
Common mistakes
Saying “correlation” for any relationship. r describes straight-line association between two quantitative variables. There is no r for eye colour and favourite sport.
Reading r near 0 as “no relationship”. There may be a strong curve. Look at the graph.
Forgetting the hat. The line gives ŷ, a prediction, not y. Write “predicted height = 34 + 0.8 (arm span)”.
Getting the residual backwards. Residual = actual − predicted = y − ŷ. A positive residual means the point is above the line.
Using b = r · sx ⁄ sy. The SD of y goes on top: b = r · sy ⁄ sx.
Extrapolating. Predictions far outside the observed x-values are guesses about a pattern you never saw.
Calling r² a percentage of points. r² = 0.81 means 81% of the variation in y is explained by the line, not that 81% of points are on it or that predictions are 81% right.
Treating correlation as cause. Ask what lurking variables could explain it, and whether the data came from an experiment.
Practice
People behind the ideas
1805 Adrien-Marie Legendre publishes the first clear explanation of the method of least squares, in a work on the orbits of comets. Robert Adrain in America finds it independently in 1808.
1809 Carl Friedrich Gauss publishes least squares too, connects it with the normal distribution, and claims he had used it since 1795, which starts a famous dispute with Legendre. Gauss had used his methods to predict where the newly found dwarf planet Ceres, discovered by Giuseppe Piazzi on 1 January 1801, would reappear.
1844 Auguste Bravais presents a formula equivalent to the correlation coefficient (his memoir is printed in 1846), decades before it gets Pearson's name. Pearson credited him in 1896, but in 1920 called that credit “hopelessly incorrect”: Bravais was studying errors in locating a point, not the link between two variables.
1886 Francis Galton publishes “Regression towards mediocrity in hereditary stature”. Children's heights deviate from the average less than their parents' do, and his word “regression” is now the name of the whole method. Galton also used these ideas to promote eugenics, a movement now rejected as scientifically wrong and harmful.
1895 Building on Galton's idea, Karl Pearson develops the correlation coefficient in the form we use today, in his paper “Notes on regression and inheritance in the case of two parents”. It is still called Pearson's r.
1973 Francis Anscombe publishes “Graphs in Statistical Analysis” with his quartet of data sets, a lasting argument for drawing the graph before trusting the numbers.
Quiz
Ten questions. Pick an answer for each, then check.
Talk about it
- Find a news headline that says one thing “is linked to” another. Is it from an experiment or from observation? What lurking variables could explain it?
- Why do you think statisticians square the residuals, rather than just adding up how far each point is from the line?
- A school finds that students who eat breakfast get higher grades. Should it make breakfast compulsory? What would you want to know first?
- When is it fair to leave an outlier out of a regression? When is it cheating?
- Smoking was shown to cause lung cancer mostly without experiments on people. How could observational evidence become convincing?
Sources
- Franz H. Messerli, “Chocolate Consumption, Cognitive Function, and Nobel Laureates”, New England Journal of Medicine 367(16), 1562–1564 (18 October 2012), doi:10.1056/NEJMon1211064: 23 countries, r = 0.791 (P < 0.0001), r = 0.862 without Sweden, and about 0.4 kg per person per year for each extra laureate.
- F. J. Anscombe, “Graphs in Statistical Analysis”, The American Statistician 27(1), 17–21 (1973), doi:10.1080/00031305.1973.10478966; data and summary statistics as listed in Wikipedia, Anscombe's quartet.
- Wikipedia, Pearson correlation coefficient: development by Karl Pearson from Galton's idea, Bravais's formula (presented 1844, printed 1846), Pearson (1895) in Proceedings of the Royal Society of London 58, 240–242, the z-score formula and the properties listed above.
- Karl Pearson, “Notes on the History of Correlation”, Biometrika 13(1), 25–45 (1920), doi:10.1093/biomet/13.1.25: Pearson withdraws his earlier credit to Bravais (pp. 28–29: “nearly the whole of the above statements are hopelessly incorrect”).
- Francis Galton, “Regression towards mediocrity in hereditary stature”, Journal of the Anthropological Institute 15, 246–263 (1886), doi:10.2307/2841583; see Wikipedia, Regression toward the mean.
- Wikipedia, Least squares: Legendre (1805), Adrain (1808), Gauss (1809) and the priority dispute, Ceres, and how to read a residual plot.
- Wikipedia, Women's Health Initiative; Writing Group for the Women's Health Initiative Investigators (J. E. Rossouw and others), “Risks and Benefits of Estrogen Plus Progestin in Healthy Postmenopausal Women”, JAMA 288(3), 321–333 (2002), doi:10.1001/jama.288.3.321: 16,608 women aged 50 to 79, stopped early after an average of 5.2 years, coronary heart disease hazard ratio 1.29 (nominal 95% CI 1.02 to 1.63).
- US Preventive Services Task Force: Postmenopausal hormone prophylaxis (1996), grade “B” to counsel women about the possible benefits and risks; Annals of Internal Medicine 137(10), 2002, grade “D” against routine estrogen plus progestin for the prevention of chronic conditions; Hormone therapy: primary prevention of chronic conditions (2022), grade “D”, consistent with 2017.
- All other data sets on this page (study hours and quiz scores, arm spans and heights, children's growth, ice cream and drownings, and the data in every interactive) are made up for teaching. All r, slope and r² values were computed exactly and rounded.