Browse

Claims

Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.

15 claims

Methods and metrics for measuring AI system capabilities, including benchmark design, evaluation validity, and interpretation of measured performance as evidence about what systems can do. Excludes evaluations of AI's effects on workplace productivity or economic outcomes.

OpenAI's intervention, process, and outcome-property evaluations validly measure chain-of-thought monitorability.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Chain-of-Thought Monitorability OpenAIimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In synthetic tasks with learned atomic skills, outcome-reward RL, but not rejection fine-tuning, generalizes to deeper unseen compositions.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Reinforcement Learning with Verifiable Rewards Artificial intelligence Large language modelsimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Results from synthetic controlled tasks on RL skill acquisition in language models generalize to natural reasoning domains.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models Reinforcement Learning with Verifiable Rewards Reasoning versus search in LLMsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

RL on the SynLogic synthetic logic dataset improves models' scores on held-out mathematics benchmarks such as AIME and MATH-500.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →LLM Reasoning Training Large language modelsimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Measured hint-verbalization rates in chain-of-thought faithfulness evaluations depend heavily on the LLM judge used.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Chain-of-Thought Monitorabilityimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Large-k pass@k is a valid measure of the boundary of language-model reasoning capability.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models Reasoning versus search in LLMsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

At large k, pass@k on tasks with small discrete answer spaces credits correct answers regardless of reasoning quality.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Reasoning versus search in LLMsimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

ProRL-trained Nemotron-Research-Reasoning-Qwen-1.5B solves Reasoning Gym tasks where DeepSeek-R1-Distill-Qwen-1.5B scores zero at pass@256.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models DeepSeek Modelsimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

DeepSeek-R1-Distill-Qwen-1.5B's near-zero Reasoning Gym scores largely reflect answer-format noncompliance rather than inability to reason.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →DeepSeek Models Large language modelsimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In Young (2026), 11 of 12 tested open-weight reasoning models acknowledged answer-changing hints in over half of cases, averaging about 70%.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Large language modelsimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Ground-truth labels in monitorability evaluations are unreliable due to intervention-evaluation filtering and outcome-property label noise.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Chain-of-Thought Monitorabilityimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

OpenAI's monitorability evaluation suite detects reduced monitorability after RL training to obfuscate chain of thought.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Chain-of-Thought Monitorability OpenAI LLM Reasoning Trainingimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Natural mathematics and coding tasks do not decompose into cleanly separable atomic skills as synthetic tasks do.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Compositional Generalization in Language Models AI Coding Benchmarksimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In Chen et al. (2025), Claude 3.7 Sonnet and DeepSeek R1 verbalized used prompt hints in about 25% and 39% of cases, respectively.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →DeepSeek Models Large language models Chain-of-Thought Monitorabilityimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In OpenAI agentic coding experiments, chain-of-thought monitors detected about 95% of reward hacks, versus 60% for action-only monitors.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Chain-of-Thought Monitorability OpenAI Agentic coding toolsimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Contribute

If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim entirely, propose it below. A proposal is reviewed on its merits; accepted claims are matched against the graph and enter it with their reasoning on record.