Sourceunknowntext stored

How Does Time Horizon Vary Across Domains?

Author not recordedMETR BlogPublished Jul 14, 2025Original︎ ↗Archived copy

Claims1

ExaminationsNone

Text84 sections

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 11, 2026

ReachNo copies recorded

StructureIn 1 claim's map0 sources downstream across 1 claim

EvidenceNot yet read

ReviewsNot reviewed

Examinations

None recorded. An examination is commissioned by a claim’s Steward for what its claim rests on; a review of the document for its own sake is asked for as a request bearing on it.

Text149 passages · 1 marked

No examination yetMarked passage

Our Work0 passages

Research0 passages

Notes0 passages

Updates0 passages

Risk Assessment0 passages

About0 passages

Donate0 passages

Careers0 passages

Search0 passages

Our Work0 passages

Research0 passages

Notes0 passages

Updates0 passages

Risk Assessment0 passages

About0 passages

Donate0 passages

Careers0 passages

Menu2 passages

CONTRIBUTORS0 passages

Thomas Kwa1 passage

Vincent Cheng0 passages

DATE0 passages

July 14, 20250 passages

SHARE0 passages

Copy Link0 passages

Citation2 passages

Copy0 passages

Subscribe1 passage

Summary4 passages · 1 marked

Software and reasoning domains like scientific QA ( GPQA ), math contests ( MATH , Mock AIME ), semi-realistic software ( METR-HRS ), and competitive programming ( LiveCodeBench ) all have 50-200+ minute horizons which are currently doubling every 2-6 months, so the ~100 minute time horizons and ~4 month doubling times seen on METR-HRS in the original paper are likely not an outlier . Visual computer use (OSWorld, WebArena) time horizons are 40-100x shorter than the software and reasoning cluster but are increasing at similar rates. Self-driving (Tesla FSD) is improving more slowly at around 0.6 doublings / year. In video understanding (VideoMME, CGBench), models are capable of answering questions on ~1 hour videos with >50% success rates, but time horizon may not be an appropriate metric. In no domain we examined is progress clearly sub-exponential.

AI capabilities improve faster in domains with automatically verifiable rewards than in other domains.Supportedmentions

Not yet read by this claim’s Steward.

Methodology17 passages

Benchmarks16 passages

Results0 passages

Trends on other domains1 passage

Main takeaway1 passage

More observations9 passages

Soundness of the time horizon metric18 passages

Video length is a poor predictor of difficulty4 passages

SWE-Lancer task value is a poor predictor of difficulty1 passage

Robustness checks3 passages

Limitations and future work6 passages

Conclusion3 passages

Appendix0 passages

Potential future experiments3 passages

Baselines for some benchmarks that lack human time data4 passages

Details for individual benchmarks3 passages

SWE-bench Verified2 passages

OSWorld2 passages

WebArena1 passage

Mock AIME4 passages

MATH2 passages

LiveCodeBench2 passages

GPQA Diamond2 passages

Video-MME2 passages

Tesla FSD2 passages

RLBench2 passages

Effective mean task duration1 passage

Other plots0 passages

Raw data3 passages

Footnotes13 notes

Cite2 passages

Subscribe0 passages

Want to contribute to this work? METR is hiring: View open roles0 passages

Featured research1 passage

Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity1 passage