The argument from information lost by a one-point summary
The argument
A 50% horizon is one point read off a fitted success-versus-task-length curve. Forecasts of when agents can do economically or strategically relevant work need the success rate at the reliability that work demands (METR's own 80% horizons are about 4 to 6 times shorter), its dependence on scaffolding, inference budget and repeated attempts, and how it transfers beyond one software task suite. These vary across models and domains and are not recoverable from the p50 figure alone, and the p50 estimate itself becomes noisy at long durations (long p50 horizons on the current suite are unreliable measurements). So a single p50 number leaves out information a forecast needs.
Does it work?
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that forecasts need reliability-, protocol- and domain-specific information that varies independently of the p50 figure, the conclusion that one p50 number is an inadequate summary follows for most forecasting purposes. The premise about 80% horizons being several times shorter is directly reported by METR; the premise that p50 is noisy at long durations rests on linked claim, not yet assessed. The inference is weaker for forecasts that target only growth rates, where the parallel-trend counterargument bites.
Premises
- UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →METR Time Horizon 1.1's 50% time horizons above roughly 16 human-hours are unreliable measurements.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
No source is recorded as making it.