The argument from observed transfer to real R&D work
The argument
METR's high-access evaluation reports, beyond the benchmark trend, staff-reported uplift and redacted internal evidence that the R&D process itself has sped up, with a majority of the speed-up attributed to AI use. Software engineering is a large part of AI R&D, and benchmark software trends transferring to real software work would carry much of the gain across. Other cheaply verifiable research domains (mathematics, vulnerability discovery) have already fallen to AI. Together these indicate benchmark gains are carrying over to AI R&D rather than staying confined to benchmarks.
Does it work?
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 inconclusive
Granting its premises, that the R&D process itself has measurably sped up through AI use and that software gains transfer to real work, non-generalization would be unlikely. The weight falls on premises that are weakly established: the internal evidence is largely redacted and self-reported, and transfer of benchmark software trends to real software work is itself unsettled.
Premises
- InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →Time-horizon trends measured on benchmark software tasks generalize to real-world software work.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
No source is recorded as making it.