Sampling and Surveys

Dewey defeats Truman, 1948

On election night in November 1948, the Chicago Daily Tribune went to press with the headline DEWEY DEFEATS TRUMAN. The paper printed about 150,000 copies. The polls had agreed: Gallup's final survey, taken October 15–25, had Thomas Dewey at 49.5% and President Harry Truman at 44.5%.

Truman won. In the official state returns he took 49.6% of the popular vote (24,179,347 votes) to Dewey's 45.1%, and 303 electoral votes to Dewey's 189 (Strom Thurmond won 39). A photo of a grinning Truman holding up the wrong headline became one of the most famous pictures in American politics.

What went wrong? Two things, and both are in this unit.

• How people were chosen. The big polls of 1948 (Crossley, Gallup and Roper) used quota sampling. Each interviewer had to find so many men and women, so many older and younger people, and so on, but chose which people to interview. That freedom let the interviewers' own choices creep in, and their samples leaned toward Dewey. The Social Science Research Council's 1949 inquiry into the polls (Mosteller and colleagues) named this interviewer choice as the main weakness of quota sampling.
• When they stopped asking. The pollsters stopped polling a week or more before the vote, but many voters decided late. Gallup's son Alec later said the main problem was quitting too soon. Gallup switched to random area probability sampling in 1950.

Both mistakes produce bias: an error that pushes results the same way every time and doesn't shrink when you ask more people.

Populations, samples and censuses

The population is the whole group you want to learn about: all voters in a country, all phones a factory made today, all students in a school. A sample is the part of the population you actually measure. A census tries to measure every member of the population. The US Census, every ten years, tries to count every person living in the country.

Why not always take a census? It can be too slow, too expensive or simply impossible. Some measurements destroy what they test: a company that crash-tested every car it built would have none left to sell. A well-chosen sample is faster and cheaper, and it can even be more accurate than a rushed census, because the work can be done carefully.

As in Unit 4, a number that describes the population is a parameter (usually unknown), and a number calculated from a sample is a statistic. If a random sample of 1,000 voters has 52% in favor, the 52% is a statistic, p̂. The share of all voters in favor is the parameter, p.

The list you actually draw the sample from is the sampling frame: a register of voters, a school's enrollment list, a map of addresses. If the frame leaves out part of the population, no amount of careful random choosing can put them back.

Random sampling methods

A sample is random (statisticians also say probability sample) when chance decides who is in it, and every member of the population has a known chance of being chosen. Chance can't play favorites, so random samples are free from the selection bias that sank the 1948 polls. There are several ways to use chance.

Simple random sample (SRS)

Every group of n members has the same chance of being the sample. Number everyone in the frame, then use a random number generator (or names drawn from a well-mixed hat) to pick n different numbers. An SRS is the baseline that the formulas in Unit 5 assume.

Stratified random sample

Split the population into groups called strata whose members are alike in some way that matters (grade level, region, age group), then take a separate SRS inside each stratum. Stratifying guarantees every group is represented, and when the strata really differ, the estimate is more precise than an SRS of the same size.

Cluster sample

Split the population into clusters, usually by location (homerooms, city blocks, crates in a warehouse), pick some clusters at random, and measure everyone in the chosen clusters. It is cheap: an interviewer can visit one block instead of houses all over town. The catch: if each cluster is full of similar people, a few clusters give a noisy estimate.

Systematic sample

To pick n from a list of N, work out the gap k = N ÷ n, choose a random starting point among the first k names, then take every k-th name after it. It's quick to do by hand. It goes wrong if the list has a pattern that repeats every k places, such as a rota where every 7th entry is a Sunday.

Multistage sample

Large surveys combine methods in stages. A national survey might pick counties at random, then neighborhoods at random inside the chosen counties, then households at random inside those neighborhoods. Each stage uses chance, so the whole sample is still random, but interviewers only travel to a few places.

Compare the methods

This made-up town has 200 households on 20 streets, split into four neighborhoods. Each household says whether it would use a new bus route. Neighborhood A, next to the bus depot, is keen; D, on the far side, is not. Overall, 90 of 200, or 45%, say yes. Take samples of 20 households with each method and watch where the estimates land.

Press 100 with every method and compare the rows. What you should see:

• Every random method is centered on the truth. None is biased.
• Stratified and systematic tie for the smallest spread: an SD of about 9.2 points, against about 10.6 for simple random. Both spread the sample across all four neighborhoods, which are very different. Stratified takes 5 from each by design; systematic does it because house numbers run street by street, so every 10th house means one house on every street.
• Cluster has the widest spread. Two blocks of 10 come from at most two neighborhoods, and blocks are like their neighborhoods. Multistage (one random block per neighborhood, then 5 houses from each) fixes that.
• Convenience (houses on the three streets nearest the depot) is steady but wrong: it lands near 80% again and again. Repeating a biased method just repeats the bias.

Bad ways to sample

A convenience sample takes whoever is easiest to reach: friends, the people in the front row, shoppers at one mall at lunchtime. A voluntary response sample lets people choose themselves: call-in shows, website polls, “text YES to…”. Both let something other than chance decide who answers.

Voluntary response is especially bad because the people who bother to reply usually feel strongly, and often negatively. A website poll can collect thousands of votes and still tell you nothing about the population.

Kinds of bias

A sampling method is biased if, used again and again, it would consistently overestimate or underestimate the parameter. Random sampling removes one source of bias, the choosing, but a survey can still go wrong in other ways.

Undercoverage and nonresponse are both about who ends up in the data. Response bias and wording effects are about what they say. A survey can suffer from several at once.

Bigger isn't better: the 1936 Literary Digest poll

In 1936 the magazine The Literary Digest mailed mock ballots to about 10 million Americans. The names came from its own readers, car registrations and telephone lists. About 2.38 million sent them back, one of the largest polls ever. It predicted that Alf Landon would beat President Franklin Roosevelt with 57% of the vote and 370 electoral votes.

Roosevelt won with 60.8% of the vote and every state except Maine and Vermont (523 electoral votes). George Gallup's much smaller poll, about 50,000 people, picked Roosevelt to win, with 55.7% of the vote.

The Digest's sample had two problems. Its lists leaned toward better-off households (undercoverage), and the 2.38 million who chose to reply were not like the 7.6 million who didn't (nonresponse). A 1988 analysis concluded that nonresponse was the bigger problem. The magazine folded in 1938.

Wording and question order

Even with a perfect random sample, the answers depend on the question. Pew Research Center tests this with split-ballot experiments: half the sample, chosen at random, gets one version of a question and half gets another. Because chance decided who got which, any big difference comes from the wording. Here are four real examples.

Common ways a question pushes people:

• Leading or loaded words: “Do you support the reckless plan…?”, or opening with a reason to agree.
• Double-barreled questions ask two things at once: “Is the cafeteria food tasty and healthy?” What should someone answer if it's tasty but greasy?
• Agreeing to be agreeable: people tend to say “agree” to agree–disagree statements, whichever way they're worded. Asking people to choose between two balanced statements avoids that.
• Answer choices: the list of options you offer shapes the answers, and a missing option can't be chosen.
• Order: earlier questions set the mood for later ones, and the order of the options can matter too. Good surveys randomize the order of options where they can.
• Wanting to look good: on sensitive topics, people give the answer they think sounds best. Private, anonymous formats help.

Designing a survey

Good questions are short, neutral and specific, ask one thing at a time, offer balanced answer choices (including a way to say “not sure” when that's honest), and use words everyone understands.

Before the real survey, run a pilot test (or pretest): try the questions on a small group from the population and see where people hesitate, misread or skip. Professional pollsters also use focus groups and one-on-one interviews in which people say aloud what they think a question means.

Experiments vs observational studies

A survey observes. It records what people think or do, without changing anything. Questions about cause and effect need something more.

• In an observational study, researchers measure variables but don't try to influence them. Comparing the grades of students who choose to join a study group with those who don't is observational.
• In an experiment, researchers deliberately impose a treatment on the subjects and measure the result. Assigning students to study groups and comparing their grades is an experiment.

The variable the researchers change is the explanatory variable, or factor; its settings are levels. The outcome they measure is the response variable. The people (or plants, or batches) receiving treatments are the subjects or experimental units. With more than one factor, each treatment is a combination of levels: 3 fertilizer amounts × 2 watering schedules = 6 treatments.

Observational studies can show association, as in Unit 6's chocolate and hormone therapy examples. People who chose differently also differ in other ways, so the effect of the explanatory variable can't be untangled from those lurking variables. When two variables' effects on the response can't be told apart, they are confounded.

Designing an experiment

Good experiments share four principles.

1. Comparison. Compare two or more treatments, often a new treatment against a control group that gets the usual treatment, nothing, or a placebo. Without a comparison group you can't tell what would have happened anyway.
2. Random assignment. Use chance to decide which subjects get which treatment. That makes the groups alike on average in everything, including lurking variables nobody thought to measure.
3. Control. Keep everything else the same for all groups: same instructions, same timing, same measuring tools.
4. Replication. Use enough subjects in each group that chance differences between them even out. (Repeating a whole study with new subjects is also called replication, and it matters just as much.)

Placebos and blinding

People often feel better just because they are being treated. This is the placebo effect. A placebo is a dummy treatment, like a sugar pill or a saltwater shot, that looks the same as the real thing, so the control group has the same experience apart from the treatment itself.

Expectations affect researchers too. In a single-blind experiment, either the subjects or the people measuring the response don't know who got which treatment. In a double-blind experiment, neither do.

Why assign at random?

Twenty volunteers will test a new warm-up routine against the usual one. Eight of them are varsity athletes, who will run fast whatever warm-up they do. Compare three ways to make the groups.

When volunteers choose, the athletes tend to pick the exciting new routine. If the new group runs faster, is that the warm-up or the athletes? You can't tell: the treatment and athletic ability are confounded. Random assignment doesn't make the groups identical, but it makes any imbalance a matter of chance, which is exactly what the significance tests in Unit 5 account for.

Blocking and matched pairs

If you already know a variable affects the response, don't leave it to chance. Group similar subjects into blocks (athletes and non-athletes, say), then randomly assign treatments within each block. This is a randomized block design. It does for experiments what stratifying does for samples: it removes a known source of variation, so the treatment effect is easier to see.

A matched pairs design is the smallest kind of block: pairs of very similar subjects, such as twins, with a coin flip deciding which one gets which treatment. Or each subject gets both treatments, in a random order, and acts as their own control: for example, each runner tries both warm-ups on different days, with a coin flip deciding which comes first.

A real trial: the 1954 polio vaccine

In the early 1950s, polio was one of the most feared childhood diseases in America. In 1954 Jonas Salk's new vaccine was tested in one of the largest medical experiments ever, evaluated by Thomas Francis Jr.'s team at the University of Michigan: 1,829,916 children in 211 areas of 44 states were included. It was really two studies, side by side.

• Observed-control areas: second graders whose parents agreed got the vaccine; first and third graders were watched as a comparison group, with no placebo.
• Placebo-control areas: children in grades 1–3 whose parents agreed got shots of either vaccine or an identical-looking placebo, by chance, and neither families nor doctors knew which. Doctors also judged whether a child had polio without knowing what the child had been given. It was a randomized, double-blind experiment.

In the placebo-control areas, the vaccinated children had about 16 cases of paralytic polio per 100,000, against 57 for the placebo group. That difference, between two groups made alike by chance, was the evidence that the vaccine worked.

Look at the third bar. The children whose parents said no had a lower rate (36) than the placebo children (57), even though neither group was protected. The volunteers' families were different: the trial's report found striking differences in income and education between families who joined and those who didn't. That's why comparing vaccinated volunteers against everyone else, as the observed-control areas did, was a weaker design: 17 cases per 100,000 among vaccinated second graders against 46 among first and third graders. That comparison mixes the vaccine's effect with grade level and family background. The randomized half of the trial avoided that problem.

Scope of inference

Two different uses of chance give two different kinds of conclusion.

• Random sampling lets you generalize from the sample to the population it was drawn from.
• Random assignment lets you conclude that the treatment caused the difference in the response.

Most experiments use volunteers, so they land in the bottom-left cell. The polio trial randomly assigned children whose parents volunteered, so strictly its causal conclusion was for children like them. Doctors then had to judge that the vaccine would work the same way in other children too, and later experience bore that out.

When the response is a category rather than a number, such as “injured or not”, the groups in an experiment like this are compared with a chi-square test, the subject of Unit 8.

Common mistakes

Practice

History

• 1747 Aboard HMS Salisbury, naval surgeon James Lind gives 12 sailors with scurvy six different treatments, two sailors each. The pair given oranges and lemons do best. It is one of the earliest planned comparisons of treatments, though not randomized.
• 1934 Jerzy Neyman shows that random (stratified) sampling is more reliable than choosing a sample “purposively” to look like the population, and how to measure its accuracy.
• 1935 R. A. Fisher's book The Design of Experiments makes the case for random assignment, replication and blocking. (Fisher's “lady tasting tea” opens Unit 5.)
• 1936 The Literary Digest's 2.38-million-reply poll calls the election for Landon; Gallup's much smaller sample picks Roosevelt.
• 1948 Quota-sampled polls call the election for Dewey. The same year, the British Medical Research Council publishes a trial of streptomycin for tuberculosis that used random numbers in sealed envelopes to decide who got the drug, widely called the first modern randomized controlled trial.
• 1954 The Salk polio vaccine trial includes a randomized, double-blind, placebo-controlled experiment with about 400,000 children in its vaccine and placebo groups.

Quiz

Ten questions. Pick an answer for each, then check.