IDEA FOUND // IDEA 124
Robustness Checks
“A conclusion that vanishes under one reasonable choice deserves less confidence than one that survives several.”
01 / PLAINLY
What it means, plainly
A robustness check changes analysis choices that could also be defended and sees whether the conclusion holds. It shows how the result depends on the variants that were actually tested.
02 / CONTEXT
A little more
Robustness checks examine whether a result changes across defensible specifications, exclusions, or coding choices. They reveal which decisions carry the conclusion but cannot replace sound design.
03 / WHY IT MATTERS
Why it matters
It separates a stable pattern from one that appears only under a particular cutoff, exclusion, or coding choice.
04 / EXAMPLE
A familiar example
An effect appears only after excluding a group at an arbitrary threshold and fades under other plausible cutoffs.
05 / LIMIT
What it does not mean
Trying many variants and publishing only the favorable one is selective search, not robustness.
06 / NOTICE
Notice it in your day
Take a short list of numbers and compute its mean with and without the most extreme value. Describe how much the summary depends on that choice.
FINAL NOTE
The idea worth keeping
Inspect the full set of reasonable choices and the reason for each one.
QUESTIONS / 02
Questions people still have
What does a robustness check vary?
Specifications, exclusions, or coding choices that can also be reasonably defended.
Can robustness checks replace good study design?
No. They only reveal how the result depends on analysis choices.