Claims
Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.
KindShowingImportancePrizesTopicMMLU Benchmark
Claims about the Massive Multitask Language Understanding (MMLU) benchmark, including its design, evaluation results, and how model prompting or training affects performance on it. Excludes other language-model benchmarks.
METR and MMLU version-linking bridges with six or seven common systems have about 80% power for 25% slope departures.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Capability EvaluationMETR Time Horizon Benchmarkimportance · minorImportance 0.35, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Qwen-2.5-7B-Instruct and Phi-3.5-mini-instruct fail explicit sandbagging compliance gates on MMLU-Pro.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →AI SandbaggingAI Capability Evaluationimportance · settledImportance 0.15, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Normalized entropy of Llama 3 8B’s answer-position distribution on MMLU-Pro under an explicit sandbagging prompt
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →Large language modelsAI SandbaggingMultiple-Choice Answer Position Biasimportance · minorImportance 0.25, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Share of MMLU-Pro items on which sandbagging Llama 3 8B gives the same answer letter after a cyclic option shift under greedy decoding
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →AI SandbaggingMultiple-Choice Answer Position BiasAI Capability Evaluationimportance · minorImportance 0.25, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Jensen-Shannon divergence between Llama 3 8B's MMLU-Pro sandbagging response-position distributions under original and cyclically shifted option orders
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →QuantityA quantity: a value the graph estimates rather than a proposition it affirms or denies. Its estimate stands where a claim's status would.constitution →Large language modelsAI SandbaggingMultiple-Choice Answer Position Biasimportance · minorImportance 0.25, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Recovering most password-locked model performance on MMLU without chain-of-thought requires a few hundred examples.
UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Large language modelsLatent Knowledge in Language ModelsLLM Prompt Sensitivityimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →
Contribute
If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim or a question entirely, propose it below. A proposal is reviewed on its wording; accepted proposals are matched against the graph before they enter it.