Claims
Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.
KindShowingImportancePrizesTopicTest representativeness
Claims about whether test conditions, test data, or evaluation environments are representative of operational/real-world conditions, and how such representativeness assumptions determine the evidential weight of test data for inferring operational performance, reliability, or risk. Domain-general; AI-specific benchmark/external-validity questions belong to their own tags.
Expert performance on isolated occupational tasks is insufficient to establish expert performance of the occupation.
SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →evaluativeA judgment of worth or quality against some standard: good, fair, effective.constitution →AI Capability Evaluationimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
GDPval tasks are based on actual work products produced by experienced industry professionals.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →AI Capability Evaluationimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
METR's task suite, including its messiest tasks, is much less messy than typical real-world work.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Coding BenchmarksAI Capability Evaluationimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
AI scheming evaluations use contrived scenarios that strongly cue misaligned actions.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI SchemingAI Capability Evaluationimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Hint-style reward hacks planted in controlled experiments represent reward hacks learned in realistic reinforcement learning.
DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Reward Hackingimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
The evidential weight of test data for operational risk depends on assumptions about test representativeness
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Safety assuranceimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Contribute
If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim or a question entirely, propose it below. A proposal is reviewed on its wording; accepted proposals are matched against the graph before they enter it.