Multiple Comparisons

“The more doors you try, the more likely one opens by chance alone.”

01 / PLAINLY

What it means, plainly

When many hypotheses are tested, there are more chances to obtain at least one extreme result even if the null hypotheses hold. Different procedures control different versions of this problem.

02 / CONTEXT

A little more

Testing many hypotheses raises the chance of at least one extreme result under the nulls. Procedures such as Bonferroni, Tukey, or false-discovery control serve different goals.

03 / WHY IT MATTERS

Why it matters

It prevents one striking finding from being highlighted without revealing how many tests had a chance to produce it.

04 / EXAMPLE

A familiar example

With twenty independent tests at the 0.05 level, the chance of at least one false positive is far above 5%.

05 / LIMIT

What it does not mean

There is no universal correction; controlling a family of errors differs from controlling the expected false-discovery proportion.

06 / NOTICE

Notice it in your day

Imagine checking twenty possible relationships and showing only the most striking one. Write down what fact about that search should accompany the result.

FINAL NOTE

The idea worth keeping

Count how many opportunities the analysis had to find something striking.

QUESTIONS / 02

Questions people still have

Why do many tests raise false-positive risk?

Each test adds another opportunity for an extreme result under the null hypotheses.

Is there one correction for every situation?

No. The method depends on which errors or false discoveries need to be controlled.