Step 13: assistant

Calls update_claim_assessment

{
  "status": "supported",
  "claim_id": "50758adb-ea92-4663-adb8-70d8985d4ebd",
  "assessment": "The 50% task-completion time horizon, popularized by METR, is the human task length at which an agent is predicted to succeed half the time on a particular suite of mostly software tasks. The case that this single figure is an inadequate summary for forecasting is well supported, and is acknowledged in part by METR itself, which reports 80% horizons alongside it and notes that they are roughly four to six times shorter. Forecasts of when agents can do real work need success rates at the reliability that work demands, the effect of scaffolding, inference budgets and repeated attempts, and evidence that the suite generalizes to other domains; none of these can be read off one p50 number, and the p50 estimate itself becomes poorly constrained at long durations. Independent methodological audits, such as a 2026 preprint on frontier-AI forecasting measurement, reach the same conclusion and propose reliability curves across duration, attempts and budgets in its place.\n\nThe strongest counterpoint is that METR found the 80% horizon doubling at essentially the same rate as the 50% horizon. If the reliability curve shifts in parallel, the p50 trend is an adequate proxy for the rate of progress, though not for the level of capability at deployment-relevant reliability. The claim therefore holds most clearly for forecasts of absolute capability or deployment readiness, and more weakly for forecasts only of growth rates. Whether the parallel-shift pattern persists as models and task suites change is the main open question.",
  "confidence": 0.72,
  "marginal_yield": 0.3,
  "reasoning_trace": "Own model: p50 is one quantile of a fitted logistic over task length on one task suite; any forecast whose target is \"reliably does X\" or \"does X in another domain\" needs more than that point unless the curve shape and domain mapping are stable and known. Evidence: Kwa et al. 2025 (arXiv 2503.14499, sec 3.2.1) report 80% horizon doubling time 204 days vs 207 for 50%, with 80% horizons 4-6x shorter, and say models \"cannot reliably perform tasks of moderate length\". This both supports the claim (level at useful reliability differs greatly from p50) and grounds the main counter-argument (rates are parallel). Costa 2026 (arXiv 2608.14903) audits the forecasting record: only seven systems jointly observe compute and a METR p50, TH1.0-to-1.1 linking slope 1.206, 73% of quantitative events from one programme; it lists one-point scores as an audit failure and recommends reliability curves over duration, attempts and budget, while finding no replacement scalar. The linked subclaim that long p50 horizons on TH1.1 are unreliable (unassessed) would further support the claim. Status supported rather than verified because \"inadequate\" is evaluative and depends on forecasting purpose; for pure rate-of-progress extrapolation the parallel-trend evidence gives p50 a defensible role. Credence omitted as the claim is evaluative."
}

Calls evaluate_argument

{
  "verdict": "likely",
  "decisive": false,
  "evaluation": "Granting that forecasts need reliability-, protocol- and domain-specific information that varies independently of the p50 figure, the conclusion that one p50 number is an inadequate summary follows for most forecasting purposes. The premise about 80% horizons being several times shorter is directly reported by METR; the premise that p50 is noisy at long durations rests on [[claim:1d21c694-7f35-4518-ba59-770d0cfef986]], not yet assessed. The inference is weaker for forecasts that target only growth rates, where the parallel-trend counterargument bites.",
  "argument_id": "85d9ddb6-6478-488a-b112-491af2d834a1"
}

Calls evaluate_argument

{
  "verdict": "slight",
  "decisive": false,
  "evaluation": "Granting that the 50% and 80% horizons grow in parallel, p50 is an adequate proxy for the rate of progress, but the claim concerns forecasting capability generally, including levels at useful reliability, other domains and other protocols, where a fixed offset estimated on one suite does not carry over. The parallel trend is also measured on few models with wide error bars, so it shows less than a full summary would need.",
  "argument_id": "8296475c-d3c9-4a91-bb37-9773e6159d08"
}