Claims

Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.

16 results

Evaluation methods and benchmarks for assessing AI code generation and software development capabilities, including their validity, reliability, and correlation with real-world performance.

On Qwen3.5 CodeContests, upgrading the judge narrows debate’s advantage over consultancy by raising consultancy performance.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Large language models LLM-as-a-Judgeimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Weak LLM judges' macro-F1 improvement from proposer–critic debate over open consultancy on CodeContests+ and ARC-AGI-2

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →LLM-as-a-Judge AI Capability Evaluationimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

AI agents' success on real-world software tasks judged by maintainer review improves at a similar rate to their benchmark success.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Coding Capability Agentic coding toolsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Success probability of optimal nested scalable oversight in the Backdoor Code game at a 400-point general Elo gap

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI Controlimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Doubling time, in days, of the 50%-task-completion horizon of frontier AI models released from 2024 onward on METR's software task suite

≈ 103 daysQuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI Capability Evaluation LLM capability trendsimportance · notableImportance 0.55, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

How can measured AI productivity effects on developers be reconciled with benchmark scores and anecdotal reports of AI helpfulness and adoption?

LeaningThe evidence favors an answer without establishing it.constitution →QuestionA question: answered rather than affirmed or denied, with its candidate answers beneath it. Its standing says how well answered it is.constitution →AI Developer Productivity Self-Reported AI Productivity Estimates Generative AI adoption and diffusionimportance · notableImportance 0.55, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

On METR tasks, AI agents' success improves at a similar rate across high- and low-messiness subsets as models are released.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Capability Evaluation LLM capability trends Task Characteristics Moderating AI Gainsimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

METR's task suite, including its messiest tasks, is much less messy than typical real-world work.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Capability Evaluation Test representativenessimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

AI coding agents' benchmark-passing patches are often rejected by human maintainers.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Agentic coding tools Artificial intelligenceimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Time-horizon trends measured on benchmark software tasks generalize to real-world software work.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Capability Evaluationimportance · majorImportance 0.70, from 0 to 1 · major: real consequence within a domain, actively argued. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Doubling time of the 50%-task-completion time horizon of frontier AI models on METR's software task suite, 2019–2025, in days

≈ 203 daysQuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI Capability Evaluation LLM capability trendsimportance · majorImportance 0.65, from 0 to 1 · major: real consequence within a domain, actively argued. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

METR's human baseline times reliably measure the length of its software tasks.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →importance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

50%-task-completion time horizon of Claude 3.7 Sonnet on METR's software task suite, in minutes

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI Capability Evaluationimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

RL training on two-function compositions improves pass@k on unseen deeper compositions while base-model pass@k remains near zero.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Compositional Generalization in Language Models Large language models LLM Reasoning Trainingimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Frontier coding RL reward hacks include inspecting or modifying test harnesses or grading code.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →AI Reward Hacking Reinforcement Learning with Verifiable Rewardsimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Natural mathematics and coding tasks do not decompose into cleanly separable atomic skills as synthetic tasks do.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Capability Evaluation Compositional Generalization in Language Modelsimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Contribute

If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim or a question entirely, propose it below. A proposal is reviewed on its wording; accepted proposals are matched against the graph before they enter it.