Browse

Claims

Search the graph by meaning. Each result carries its current verdict; open one to see its decomposition, provenance, and the reasoning behind the assessment.

22 results

Claims about AI agents pursuing objectives different from the intended or training objective, particularly when learned goals generalize incorrectly to novel or shifted environments. Excludes ordinary reward hacking where an agent exploits a reward proxy without a persistent unintended goal.

Human evolution provides limited evidence about AI goal misgeneralization.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →evaluativeA judgment of worth or quality against some standard: good, fair, effective.constitution →Evolutionary Biology AI Safetyimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Goal misgeneralization is a significant failure mode in AI systems substantially more capable than current systems.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Safety AI Alignmentimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Current frontier language models exhibit goal misgeneralization distinct from specification gaming and capability failure.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language models AI Alignment AI Reward Hackingimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Training AI models with outcome-only rewards in hackable environments produces generalized misaligned reward hacking.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Reward Hacking Reinforcement Learning with Verifiable Rewardsimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Large language models trained on human data generalize intended goals well enough that goal misgeneralization is not a serious risk.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language modelsimportance · notableImportance 0.55, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Training tends to produce AI systems with context-dependent heuristics rather than unified goals that generalize beyond the training distribution.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Goal-Directedness Machine Learningimportance · notableImportance 0.60, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Sufficiently capable AI agents pursuing almost any goal have instrumental incentives to seek power and resist correction.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Alignment Faking AI Reward Hackingimportance · majorImportance 0.75, from 0 to 1 · major: real consequence within a domain, actively argued. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

InstructGPT's harmful advice reflects goal misgeneralization rather than reward misspecification in human-preference training.

DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Large language models OpenAI AI Reward Hackingimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

The indifference principle gives substantial reason to worry that future AI systems will become schemers.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →evaluativeA judgment of worth or quality against some standard: good, fair, effective.constitution →Artificial intelligence Alignment Fakingimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Indifference over training-compatible goals reliably predicts the goals acquired by SGD-trained AI systems.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Stochastic Gradient Descent Mesa-Optimization AI Goal-Directednessimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Scaling up models reduces goal misgeneralization.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Machine Learning AI Scaling Trendsimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Goal misgeneralization can arise from underspecified training data, and increasing model scale does not eliminate it.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Machine Learning AI Scaling Trendsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Goal misgeneralization can arise when correct training feedback is underspecified and consistent with multiple possible goals.

VerifiedThe evidence, examined directly, establishes the claim.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →AI Alignmentimportance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Goal misgeneralization is a plausible mechanism for catastrophic risk from powerful AI systems.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Artificial intelligenceimportance · majorImportance 0.65, from 0 to 1 · major: real consequence within a domain, actively argued. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Coherent, competent behavior from goal misgeneralization can have greater impact and cause significantly greater harm.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →importance · minorImportance 0.30, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Deep learning's inductive biases favor networks that generalize in ways humans intend.

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Artificial intelligenceimportance · notableImportance 0.60, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Goal misgeneralization persists or becomes harder to detect as AI systems become more capable.

SupportedThe evidence favors the claim, but the examination is incomplete or the evidence is indirect.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →AI Scaling Trendsimportance · notableImportance 0.60, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Deep RL agents can competently pursue unintended goals under distribution shift while retaining their capabilities.

VerifiedThe evidence, examined directly, establishes the claim.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Reinforcement Learning AI Reward Hackingimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

A PPO agent trained on CoinRun with the coin always at the level's end navigates to the end and ignores a relocated coin at test time.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Reinforcement Learningimportance · settledImportance 0.20, from 0 to 1 · settled: uncontested, so low even when much depends on it. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Claude 3 Opus attempts to exfiltrate its weights when given an easy opportunity in conflict-with-training settings.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · verifiableA factual claim that could be checked directly against observation or primary records.constitution →Alignment Fakingimportance · notableImportance 0.50, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Few-shot prompted language models can misgeneralize the intended task goal without reinforcement learning.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →empirical · derivedA factual claim that rests on inference from other evidence rather than direct observation.constitution →Large language modelsimportance · minorImportance 0.40, from 0 to 1 · minor: narrow or largely settled, cheap to get right. Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Mesa-optimizers developed through local optimization tend to become proxy-aligned before robustly aligned.

UnassessedNo current assessment. Attention goes where its expected value is highest and someone funds it; nothing has funded an assessment of this claim yet, and anyone can.constitution →causalA claim that one thing brings about another, not merely that the two go together.constitution →Mesa-Optimizationimportance · notableImportance 0.45, from 0 to 1 · notable: a contested point in a live debate (also the default before judging). Higher-importance claims are worth more to assess, so funding reaches them sooner.constitution →

Contribute

If a claim here is wrong, or missing evidence, open it: every claim page carries its own entry for challenges, evidence, and corrections. If the graph is missing a claim or a question entirely, propose it below. A proposal is reviewed on its wording; accepted proposals are matched against the graph before they enter it.