Jugami › Statistics and Probability › Probability: Chance, Counting and Bayes
Statistics · Unit 2
Probability: Chance, Counting and Bayes
Probability is the math of “how likely?”. It can't tell you what the next coin flip will be, but it can tell you, very precisely, what happens over thousands of flips.
In this unit you'll learn sample spaces and events, the complement, addition and multiplication rules, independence, counting with permutations and combinations, conditional probability from two-way tables, and Bayes' theorem in a friendly form. You'll also see the law of large numbers in action and learn why a coin is never “due”.
1654: two letter-writers and an unfinished game
In the summer of 1654, the French mathematician Blaise Pascal wrote to Pierre de Fermat, a lawyer in Toulouse who did mathematics in his spare time. In five letters they worked on two puzzles from gambling, both put to Pascal by the Chevalier de Méré, a French nobleman and gambler. The dice problem asked how many times you must throw a pair of dice before you can expect a double six. The problem of points asked: if a game of chance is stopped before anyone has won, how should the stakes be split fairly?
Their answers became the foundation of probability theory. Here is the problem of points in a simple version you can solve today.
Worked example: splitting the pot
Ana and Ben each put $10 in a pot. They flip a fair coin; heads gives Ana a point, tails gives Ben a point, and the first to 3 points wins all $20. The score is Ana 2, Ben 1 when the game has to stop. How should they split the $20?
- At most two more flips would settle it. Ana wins unless Ben wins both of them.
- P(Ben wins both) = ½ × ½ = ¼, so P(Ana wins) = 1 − ¼ = ¾.
- A fair split follows those chances: Ana gets ¾ × $20 = $15 and Ben gets $5.
Notice the idea: the split is based on what could still happen, not on the points already scored. Weighing outcomes that haven't happened yet is the whole job of probability.
Sample spaces and events
- A random process (or experiment) has outcomes we can't predict one at a time, like rolling a die.
- The sample space S is the list of all possible outcomes. For one die, S = {1, 2, 3, 4, 5, 6}.
- An event is a set of outcomes, like “roll an even number” = {2, 4, 6}.
- A probability is a number from 0 (impossible) to 1 (certain). All the outcomes in S add up to 1.
When all outcomes are equally likely, probability is just counting:
So P(even) = 3/6 = 1/2. For two dice, the sample space has 6 × 6 = 36 equally likely pairs. Six of them add to 7, so P(sum is 7) = 6/36 = 1/6.
Careful: the sums 2 to 12 are not equally likely. There's only one way to roll a 2 (1 and 1) but six ways to roll a 7. Always count the equally likely outcomes, not the labels.
The complement
The complement of A, written “not A” or Aᶜ, is everything in S that isn't in A. Since A and not A cover everything:
The complement is a shortcut for “at least one” questions. What's the chance of at least one head in 3 coin flips? Counting all the ways is slow, but “at least one head” is the complement of “no heads”: P = 1 − (½)³ = 1 − 1/8 = 7/8.
The addition and multiplication rules
“Or”: the addition rule
We subtract the overlap because it was counted twice. Draw one card from a standard 52-card deck. What's P(heart or king)? There are 13 hearts and 4 kings, but the king of hearts is in both groups: P = 13/52 + 4/52 − 1/52 = 16/52 = 4/13.
If two events can't happen together, they are mutually exclusive (disjoint), P(A and B) = 0, and you just add. P(roll a 1 or a 6) = 1/6 + 1/6 = 1/3.
“And”: the multiplication rule
P(B | A) is read “the probability of B given A”: the chance of B once you know A happened. Draw two cards without putting the first back. P(both aces) = 4/52 × 3/51 = 12/2,652 = 1/221. The second fraction is 3/51 because one ace and one card are already gone.
Independence
Events are independent when knowing one happened doesn't change the chance of the other: P(B | A) = P(B). Then the rule becomes simply P(A and B) = P(A) × P(B). Coin flips are independent: P(three heads in a row) = ½ × ½ × ½ = 1/8. Drawing cards without replacement is not independent, because the deck changes.
Disjoint is not the same as independent. If A and B are disjoint and A happens, then B can't happen, so knowing A changes B's chance to 0. Disjoint events (with nonzero chances) are always dependent.
Counting: permutations and combinations
Equally likely outcomes turn probability into counting, so we need fast ways to count.
- Multiplication principle: if one choice has m options and a second has n, together there are m × n. A 3-shirt, 4-pants wardrobe gives 12 outfits.
- Permutations (order matters): choose and arrange k of n things, nPk = n × (n − 1) × … (k factors). Gold, silver and bronze among 8 runners: 8 × 7 × 6 = 336 ways.
- Combinations (order doesn't matter): nCk = nPk ÷ k!. A 3-person committee from 8 people: 336 ÷ (3 × 2 × 1) = 56. Each committee was counted 6 times as a permutation, once for each order.
Worked example: a 6-from-49 lottery
A lottery draws 6 numbers from 1 to 49, and order doesn't matter. How many tickets are possible?
- 49C6 = (49 × 48 × 47 × 46 × 45 × 44) ÷ (6 × 5 × 4 × 3 × 2 × 1).
- That's 10,068,347,520 ÷ 720 = 13,983,816.
- With one ticket, P(jackpot) = 1/13,983,816: about 1 in 14 million.
Quick test: “Does swapping two people give a different result?” If yes (president and treasurer), use permutations. If no (two members of a team), use combinations.
Next: Counting and Combinations goes further, with factorials, Pascal's triangle, Powerball odds, the birthday problem and poker hands.
Conditional probability and two-way tables
A two-way table sorts people by two categorical variables at once. Here are 100 made-up students:
| Has a pet | No pet | Total | |
|---|---|---|---|
| Plays a sport | 45 | 30 | 75 |
| No sport | 15 | 10 | 25 |
| Total | 60 | 40 | 100 |
- P(sport and pet) = 45/100 = 0.45. Out of everyone, 45 are in that cell.
- P(pet given sport) = 45/75 = 0.6. “Given sport” shrinks the world to the 75 students who play a sport.
Check: 0.45 ÷ 0.75 = 0.6. Same answer. The formula and the “shrink the table” method always agree.
Is having a pet independent of playing a sport here? P(pet) = 60/100 = 0.6 and P(pet | sport) = 0.6. Knowing a student plays a sport doesn't change the chance, so in this table they are independent. Try the second data set in the explorer to see two variables that are not independent.
Order matters in conditional probability: P(pet | sport) = 45/75 = 0.6, but P(sport | pet) = 45/60 = 0.75. Mixing them up is one of the most common mistakes in probability, and it's the key to the next section.
Bayes' theorem with natural frequencies
A medical test is fairly accurate, yet a positive result can still mean you probably don't have the condition. The easiest way to see why is to imagine a crowd, which is called using natural frequencies.
Worked example: a positive test
A condition affects 1% of people. The test catches 90% of people who have it (sensitivity). It also wrongly comes back positive for 9% of people who don't (the false-positive rate). You test positive. What's the chance you have the condition?
- Imagine 10,000 people. 1% of them, 100, have the condition; 9,900 don't.
- Of the 100, 90% test positive: 90 true positives.
- Of the 9,900, 9% test positive: 891 false positives.
- So 90 + 891 = 981 people test positive, and only 90 of them have the condition: 90 ÷ 981 ≈ 0.092, about 9.2%.
So a positive result here is far from proof, even though the test catches 90% of real cases. The catch is that the condition is rare, so the healthy crowd is huge, and 9% of a huge crowd outnumbers 90% of a small one. That's Bayes' theorem. In symbols, P(sick | positive) = P(sick and positive) ÷ P(positive), the same conditional-probability formula you already know.
Try raising “Have the condition” to 20%. With the same test, a positive result becomes much more convincing. Doctors often order a second test for exactly this reason.
The law of large numbers and the gambler's fallacy
Flip a fair coin 10 times and you might get 7 heads. Flip it 10,000 times and the share of heads will almost surely be very close to 0.5. That's the law of large numbers: as the number of trials grows, the relative frequency of an event settles down near its probability. It's why casinos and insurance companies can plan on averages even though each bet or claim is unpredictable.
The gambler's fallacy
The law of large numbers does not say that a run of heads must be “balanced out” by tails soon. The coin has no memory: after 5 heads in a row, P(tails) on the next flip is still ½. Believing a result is “due” is called the gambler's fallacy. The share of heads gets close to ½ because the early streak gets swamped by thousands of later flips, not corrected.
The Monte Carlo fallacy. At the Monte Carlo Casino in 1913, a roulette wheel landed on black 26 times in a row. The gambler's fallacy is sometimes called the Monte Carlo fallacy after that famous run.
Puzzle: the Monty Hall problem
On a game show there are three doors. A car is behind one, goats are behind the other two. You pick a door. The host, who knows where the car is, always opens one of the other doors to show a goat, then offers you a switch. Should you switch?
Most people say it doesn't matter, 50–50. Play a few rounds, then let the simulator play hundreds.
Why switching wins 2 out of 3 times
Your first pick is right with probability 1/3, and wrong with probability 2/3. If you were right, switching loses. If you were wrong, the host must open the only other goat door, so the door left over is the car, and switching wins. So switching wins exactly when your first pick was wrong: 2/3 of the time. The key is that the host's choice isn't random: he never opens your door or the car.
Common mistakes
Adding when events overlap. P(heart or king) is not 13/52 + 4/52 = 17/52. The king of hearts was counted twice, so the answer is 16/52 = 4/13.
Multiplying when events aren't independent. P(two aces without replacement) is not (4/52)², because the first draw changes the deck. Use 4/52 × 3/51.
Flipping a conditional probability. P(positive | sick) = 90% does not mean P(sick | positive) = 90%. In the example it's only about 9%.
Thinking a result is “due”. Independent trials have no memory. After a streak of reds, the chance of black on the next spin is unchanged.
Mixing up permutations and combinations. A committee of 3 from 8 is 8C3 = 56, not 8P3 = 336, because order doesn't matter in a committee.
Practice
People behind the ideas
1563 Girolamo Cardano has probably finished his “Liber de Ludo Aleae” (“On games of chance”), one of the first studies of the chances in dice games. It isn't printed until 1663, long after his death.
1654 The Chevalier de Méré, a French nobleman and gambler, puts two problems to Blaise Pascal: a dice wager and the problem of points. In the dice question he reasoned, by an old rule of thumb (4/6 = 24/36), that betting on at least one six in 4 rolls of one die and betting on at least one double six in 24 rolls of two dice should be equally good. In play, the first bet won slightly more than half the time, while the second lost slightly more than half the time: 1 − (5/6)⁴ ≈ 0.518, but 1 − (35/36)²⁴ ≈ 0.491. He asked Pascal why. Pascal and Pierre de Fermat work the problems out in their letters and are now seen as joint founders of probability theory.
1657 Christiaan Huygens, who heard about the Pascal–Fermat work in Paris, publishes “De Ratiociniis in Ludo Aleae”, the first printed work on probability.
1713 Jacob Bernoulli's “Ars Conjectandi” (“The art of conjecturing”) is published in Basel, eight years after his death. It includes his law of large numbers.
1764 Thomas Bayes's “Essay towards solving a problem in the doctrine of chances” is published in the Philosophical Transactions of the Royal Society, three years after his death. His friend Richard Price found it among his papers and sent it in.
1913 A roulette wheel at the Monte Carlo Casino lands on black 26 times in a row, the story behind the name “Monte Carlo fallacy”.
1990 In her “Ask Marilyn” column in Parade magazine on 9 September, Marilyn vos Savant answers the Monty Hall puzzle: switch. About 10,000 readers write in, most of them telling her she is wrong. Switching wins 2 times out of 3.
Quiz
Eight questions. Pick an answer for each, then check.
Talk about it
- A weather app says “70% chance of rain” and it stays dry. Was the app wrong? How could you check whether its forecasts are any good?
- A friend has lost five games of chance in a row and says they're “due” for a win. What would you tell them?
- Why is a positive result on a screening test for a rare condition often followed by a second, different test?
- Pascal and Fermat split the pot by what could still happen. Can you think of a real situation, in sports or games, where that idea would be fair or unfair?
Sources
- MacTutor History of Mathematics, Blaise Pascal biography: five letters with Fermat in the summer of 1654, the dice problem and the problem of points.
- MacTutor, Pierre de Fermat biography: the 1654 correspondence; Pascal and Fermat as joint founders of probability.
- MacTutor, Girolamo Cardano biography: “Liber de Ludo Aleae”, probably completed by 1563 and published in 1663.
- MacTutor, Christiaan Huygens biography: “De Ratiociniis in Ludo Aleae” (1657), the first printed work on probability.
- MacTutor, Jacob Bernoulli biography: “Ars Conjectandi”, Basel, 1713, and the law of large numbers.
- MacTutor, Thomas Bayes biography: the Essay in the Philosophical Transactions, 1764, sent by Richard Price.
- Wikipedia, Gambler's fallacy: the 1913 Monte Carlo Casino run of 26 blacks.
- Richard J. Pulskamp, The Division of Stakes or The Problem of Points: the Chevalier de Méré proposed both problems to Pascal.
- Wolfram MathWorld, de Méré's Problem: de Méré as a French nobleman and gambler, and his four-rolls versus 24-throws dice question.
- Wikipedia, Monty Hall problem: vos Savant's “Ask Marilyn” column in Parade, 9 September 1990, and about 10,000 letters in reply (citing John Tierney, “Behind Monty Hall's Doors: Puzzle, Debate and Answer?”, The New York Times, 21 July 1991).
Credits: all diagrams and simulators are original and made for Jugami. The two-way tables, the medical-test numbers and the Ana and Ben game are made up for teaching; no real data are used in this unit.