Claims

Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.

22 results

Claims about using large language models to evaluate, score, rank, or compare model-generated outputs, including judge accuracy, reliability, bias, and methods that improve or measure judging performance.

MacDiarmid et al. (2025)'s misalignment judges falsely classify confused, garbled, or deflecting open-weight model responses as misaligned.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Alignment Large language modelsimportance · minorImportance 0.25, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Human judges' knowledge advantage over AI models contributed to positive results in earlier simple oversight studies.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Scalable Oversightimportance · notableImportance 0.55, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Collaborative disagreement-resolution protocols yield higher weak-judge accuracy than standard fixed-position AI debate.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →AI Debate LLM Multi-Agent Deliberationimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

AI judges given human-like personas judge debate and consultancy protocols more accurately than human or default AI judges.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Large language models AI Debate LLM Prompt Sensitivityimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Scalable oversight protocols should be judged against simple deference to the evaluated model, not only against unassisted judges.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →normativeA claim about what should be done or how things ought to be, settled by argument rather than evidence alone.constitution →Scalable Oversightimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

LLMs rate arguments opposing their prior beliefs higher in pairwise quality comparisons than belief-aligned arguments.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language modelsimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

The benefits of AI debate for human judges should not be assumed to extend to AI judges in automated oversight pipelines.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →normativeA claim about what should be done or how things ought to be, settled by argument rather than evidence alone.constitution →AI Debate Scalable Oversight AI Safetyimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Weaker AI judges gain accuracy from debate comparably to human judges.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Debate AI Capability Evaluationimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Measured emergent-misalignment rates in language models vary substantially with judge model, scoring threshold, and evaluation questions.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Emergent Misalignment AI Capability Evaluation LLM Prompt Sensitivityimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

On Qwen3.5 CodeContests, upgrading the judge narrows debate’s advantage over consultancy by raising consultancy performance.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →AI Coding Benchmarks Large language modelsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Weak LLM judges' macro-F1 improvement from proposer–critic debate over open consultancy on CodeContests+ and ARC-AGI-2

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI Capability Evaluation AI Coding Benchmarksimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Under some conditions, debate between AI models improves the accuracy of judges less capable than the debaters.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Debateimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In homogeneous LLM judge panels, deliberation outcomes track majority status rather than correctness.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models LLM Multi-Agent Deliberationimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

When an LLM judge has a wrong prior, post-debate confidence tracks its induced belief more than correctness.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Debateimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Voting and deliberation among multiple LLM judges do not significantly improve group accuracy as panel size increases.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models AI Capability Evaluationimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

In LLM judge experiments with injected priors, debate improves accuracy with correct priors and reduces it with incorrect priors.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models LLM Prompt Sensitivityimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

For factual claims, debate outperforms consultancy when judged by Claude Sonnet 4.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models LLM Prompt Sensitivity Anthropicimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Some weaker LLM judges verify critic claims less in debate than in consultancy.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Debate Large language modelsimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

On CodeContests, doubling consultancy compute with independent consultants does not eliminate debate’s advantage for weak LLM judges.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →AI Debate Scalable Oversight AI Coding Benchmarksimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Debate’s gains for weak LLM judges mainly come from rejecting incorrect proposer answers.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Debateimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

At test time, one independent critique gives weak LLM judges nearly all the benefit of multi-round debate.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models AI Capability Evaluation LLM Prompt Sensitivityimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Predicting debate’s benefit to weaker LLM judges requires more than a generator–verifier performance gap.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Scalable Oversightimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Contribute

If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim or a question entirely, propose it below. A proposal is reviewed on its wording; accepted proposals are matched against the graph before they enter it.