The argument from item selection (ecological approach)
shows With representatively sampled items, the error rate among certain answers is lower than the classic ~20%, plausibly 5 to 15%.
The argument
Classic calibration studies used experimenter-selected general-knowledge items that over-represent misleading questions. Because representative random sampling of items largely eliminates overconfidence, the share of certain answers that are wrong under representative sampling should fall well below the classic figure of about one in five, toward roughly 5 to 15 percent.
Does it work?
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting its premise, the error rate among certain answers would likely fall below the classic 20% under representative sampling. The argument stands or falls with the claim that representative sampling largely eliminates overconfidence. That claim is well supported for overall calibration but less directly documented for the top confidence category, and some studies find residual overconfidence on hard items.
Premises
- UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →Representative random sampling of general-knowledge items largely eliminates overconfidence in confidence judgments
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
No source is recorded as making it.
Earlier ratings
- Nov 2, 2026likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
If classic item sets really do over-represent misleading questions, the error rate among certain answers would likely be lower under representative sampling. That would put the population-relevant figure below the classic 20%. The inference depends on the empirical finding that random sampling of items reduces overconfidence. That finding is well replicated for overall calibration, though less directly documented for the top confidence category alone. Some studies, such as Brenner and colleagues' work, find residual overconfidence even with representative items, which limits how far the figure falls.