Sample Size and the Law of Large Numbers — Why Bigger Wins (and tiny samples Lie)

"This is the moment where the concept finally clicks: small samples are drama queens, large samples are stoic realists."

You already met two characters in our statistical cast: regression to the mean (we saw how extreme outcomes tend to be followed by more ordinary ones) and base rate neglect (people ignore context and prior probabilities). Now meet the sober, mathematical sibling who tells both to calm down: the Law of Large Numbers (LLN) — and its sidekick, sample size. These explain why big samples stabilize our beliefs and why small samples make our intuitions look like tabloid headlines.

What the Law of Large Numbers actually says (without the snooze)

Informal version: If you repeat a random process many times (coin flips, customer purchases, exam scores), the average of the results will get closer and closer to the process's true average.

• Weak LLN: As n (sample size) grows, the sample average converges in probability to the expected value.
• Strong LLN: With probability 1, the sample average converges to the expected value as n → ∞.

Micro-explanation: imagine each flip of a fair coin gives 1 for heads, 0 for tails. The true average (expected value) is 0.5. LLN guarantees that if you flip the coin thousands of times, the proportion of heads will be very close to 0.5.

Why sample size matters — the human-friendly take

• Small samples are noisy. A handful of observations can swing wildly just by chance. That’s why a local hospital reporting on rare complications in a week might look like an epidemic — until a month's worth of data smooths it out.
• Big samples reduce noise. More data means fewer freak swings. The sample mean's variability shrinks roughly like 1/√n.

Mathematical nugget (short and useful):

• If observations have standard deviation σ, the standard error of the sample mean is σ / √n.
• Double n → standard error shrinks by factor 1/√2 (~0.707). Quadruple n → halves the error.

So increasing sample size has diminishing returns (you need 4× the data to cut the error in half), but the direction is always toward more stability.

Tie-back to Regression to the Mean and Base Rate Neglect

Remember regression to the mean? That effect is most dramatic when initial observations come from small samples. A superstar test performance might be an outlier caused by luck — when you test again (or with more students), the average slides back toward the true mean.

And base rate neglect? People often treat a few vivid cases as representative of the whole. LLN says: don't. Without respecting sample size and base rates, you’ll overreact to noise.

Think Prospect Theory: we overweight rare dramatic outcomes and dislike losses more. Combine that with small-sample drama and you get viral stories: "Startup doubles revenue in month 1!" — sample size is one month; probability weighting makes us much too impressed.

Real-world analogies (because metaphors = brain glue)

• Weather vs. Climate: A freak blizzard in April (small-sample event) doesn’t change the long-term climate. LLN is climate.
• Sports hot streaks: A player hits 7/8 free throws in a practice session. Is she suddenly elite? Maybe practice sessions are small-sample theatres of luck. Over the season (big sample), the true skill shows.
• Restaurants: One five-star Yelp review is sweet; a thousand reviews create a more trustworthy average.

Quick examples and a tiny calculation

Example 1 — Coin flips:

• n = 10 flips: proportion of heads could easily be 0.2 or 0.8.
• n = 10,000 flips: proportion of heads will be very close to 0.5.

Example 2 — Exam scores:

• Imagine population SD σ = 10 points. If n = 4 students, standard error ≈ 10/2 = 5 points.
• If n = 100 students, standard error = 10/10 = 1 point. Much more reliable.

Code block (pseudo-Python):

# Simulate the Law of Large Numbers
import random
def average_of_flips(n):
    return sum(random.choice([0,1]) for _ in range(n)) / n
for n in [10,100,1000,10000]:
    print(n, average_of_flips(n))

Run it — watch the proportion of heads calm down as n gets big.

Common mistakes (and how to avoid them)

1. Treating tiny samples as decisive. If your sample is small, inflate your uncertainty.
2. Confusing variability with bias. A small sample can be wildly variable but not biased. Don’t assume systematic error when you see extreme results in little data.
3. Forgetting the standard error. Always ask: how noisy is my estimate? Sigma / √n tells you.
4. Cherry-picking extremes. Selective reporting of extreme small-sample results is why tabloids rule the internet.

Practical habit: whenever you see a headline based on a small group, mentally multiply the margin of error by a lot.

When bigger doesn't fix everything

LLN reduces random noise, but it doesn't cure bias. If your measurement process is flawed, collecting more of the same flawed data just gives you a precise-but-wrong number. Remember: precision ≠ accuracy.

Also, if the world is changing (non-stationary processes), older large samples might not reflect the present.

Quick checklist for sensible statistical thinking

• What is my sample size (n)?
• What variability (σ) should I expect? Can I estimate the standard error (σ/√n)?
• Could this extreme result be a small-sample fluke (regression to the mean)?
• Am I ignoring base rates or overweighting a dramatic outlier (base rate neglect + prospect-theory vibes)?
• Is my data biased or non-stationary?

Final takeaways — what to remember going forward

• The Law of Large Numbers is the formal reason why more data tends to give more reliable averages.
• Sample size matters: small n → big uncertainty; big n → stable estimates (but diminishing returns).
• Don’t be fooled by drama: regression to the mean and base rate neglect make small samples seem meaningful when they often aren’t.
• Mind the difference between precision and correctness. Large n reduces random error, not systematic bias.

Imagine your intuition as a gossiping friend — entertaining, quick, and biased toward drama. The Law of Large Numbers is the calm statistician who walks in, shows the receipts (big datasets), and says: "This is what’s really going on." Respect both voices: intuition helps spot patterns, but LLN tells you whether those patterns survive contact with reality.

Want one final golden line to steal?

Small samples whisper rumors. Large samples publish the book.