Step 4: tool results

get_claim

{
  "claim_id": "14228127-3c75-4f02-aff0-9253356e34f6"
}

{"claim":{"id":"14228127-3c75-4f02-aff0-9253356e34f6","kind":"claim","canonical_form":"METR's human baseline times reliably measure the length of its software tasks.","claim_type":"empirical_derived","state":"active","decomposition_status":"complete","importance":0.45,"contestation":0.45,"domains":[],"tags":[{"slug":"ai-coding-benchmarks","name":"AI Coding Benchmarks"}],"steward_state":"done","stewarded_at":"2026-11-13T00:10:30.732Z","canonical_direction_note":null,"created_at":"2026-11-01T04:15:46.299Z","updated_at":"2026-11-13T00:11:08.899Z","page_url":"https://minerval.ai/claims/14228127-3c75-4f02-aff0-9253356e34f6"},"assessment":{"id":"74f66067-f6cb-4653-a3e0-37fa5b806bca","status":"doubtful","confidence":0.6,"claim_credence":0.35,"summary":"METR denominates its time-horizon metric in the time a skilled human, new to the task's context, takes to complete each task. For short tasks this rests on measured baselines: the SWAA tasks and many HCAST tasks were timed with contracted baseliners, and the task length is the geometric mean of successful attempts. As a claim about the whole suite, however, \"reliably measure\" overstates what the baselines provide. METR's own Time Horizon 1.1 release (January 2026) says human baseline times were measured for only 5 of the 31 tasks of eight hours or more; the rest use estimates. Even where baselines exist, there are usually only one or a few per task, and conditioning on successful attempts pulls ratings toward shorter times. Baseliners also lacked context on the codebases, so their times measure \"a capable outsider,\" not an experienced maintainer.\n\nThe baselines are a reasonable, roughly order-of-magnitude yardstick for short and medium tasks. For long tasks, which set the upper end of recent horizons and the post-2024 trend, they are largely estimates and not measurements. Whether this distorts the trend's slope depends on whether the errors vary with task length, and no independent check of the estimated times has been published. Such a check, for example new baselines on the estimated 8h+ tasks, would settle the question.","reasoning_trace":"Primary evidence: METR, \"Time Horizon 1.1\" (https://metr.org/blog/2026-01-29-time-horizon-1-1/): the suite grew from 170 to 228 tasks and long tasks from 14 to 31, and \"we measured human baseline times for only 5 of our 31 long (8h+) tasks. The remainder use estimated times.\" Also, 26 tasks had their human time estimates updated, which shows the length labels shift. The original paper (Kwa et al. 2025, arXiv 2503.14499) notes that conditioning on success biases ratings toward shorter task lengths, and that baseliners were domain-knowledgeable but had no task-specific context. HCAST (metr.org/hcast.pdf) reports that the share of tasks with estimated rather than measured times rises with length, because successful long baselines are hard to collect. On balance, the measurements are reliable enough for short tasks, but the claim as worded covers the suite, and for its longest and most consequential tasks the times are mostly estimates. Hence doubtful, at moderate confidence: \"reliably\" is a matter of degree, and short-task baselines are reasonably sound. This pass did not attempt an independent quantification of estimate error.","marginal_yield":0.4,"assessed_at":"2026-11-13T00:11:08.860Z","model":"claude-opus-5-5","run_id":"1bf323d4-cd67-4628-8df7-97321b20a93a","relabeled_from":null},"coverage":{"examined":true,"on":["public_evidence"]},"since_assessed":{"recorded":true,"changed":true,"premises_rejudged":0,"premises_added":0,"premises_removed":0,"sources_added":1,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[],"requests":[],"requests_total":0,"subclaim_count":0,"instance_count":1,"source_map":null}