Run step
Step 4: tool results
read_page
Input
{
"url": "https://arxiv.org/abs/2504.18530",
"max_chars": 3500
}{"ok":true,"url":"https://arxiv.org/abs/2504.18530","text":"[2504.18530] Scaling Laws For Scalable Oversight\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search · Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2504.18530 (cs)\n[Submitted on 25 Apr 2025 ( v1 ), last revised 27 Oct 2025 (this version, v3)]\nTitle: Scaling Laws For Scalable Oversight\nAuthors: Joshua Engels , David D. Baek , Subhash Kantamneni , Max Tegmark\nView a PDF of the paper titled Scaling Laws For Scalable Oversight, by Joshua Engels and 3 other authors\nView PDF\nHTML (experimental)\nAbstract: Scalable oversight, the process by which weaker AI systems supervise stronger ones, has been proposed as a key strategy to control future superintelligent systems. However, it is still unclear how scalable oversight itself scales. To address this gap, we propose a framework that quantifies the probability of successful oversight as a function of the capabilities of the overseer and the system being overseen. Specifically, our framework models oversight as a game between capability-mismatched players; the players have oversight-specific Elo scores that are a piecewise-linear function of their general intelligence, with two plateaus corresponding to task incompetence and task saturation. We validate our framework with a modified version of the game Nim and then apply it to four oversight games: Mafia, Debate, Backdoor Code and Wargames. For each game, we find scaling laws that approximate how domain performance depends on general AI system capability. We then build on our findings in a theoretical study of Nested Scalable Oversight (NSO), a process in which trusted models oversee untrusted stronger models, which then become the trusted models in the next step. We identify conditions under which NSO succeeds and derive numerically (and in some cases analytically) the optimal number of oversight levels to maximize the probability of oversight success. We also apply our theory to our four oversight games, where we find that NSO success rates at a general Elo gap of 400 are 13.5% for Mafia, 51.7% for Debate, 10.0% for Backdoor Code, and 9.4% for Wargames; these rates decline further when overseeing stronger systems.\nComments:\n33 pages, 17 figures; The first three authors contributed equally\nSubjects:\nArtificial Intelligence (cs.AI) ; Computers and Society (cs.CY); Machine Learning (cs.LG)\nCite as:\narXiv:2504.18530 [cs.AI]\n(or\narXiv:2504.18530v3 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2504.18530\nFocus to learn more\narXiv-issued DOI via DataCite\nJournal reference:\nNeurIPS 2025 (Spotlight)\nSubmission history\nFrom: David D. Baek [ view email ] [v1]\nFri, 25 Apr 2025 17:54:27 UTC (4,030 KB)\n[v2]\nFri, 9 May 2025 16:30:59 UTC (4,176 KB)\n[v3]\nMon, 27 Oct 2025 02:48:09 UTC (1,452 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled Scaling Laws For Scalable Oversight, by Joshua Engels and 3 other authors\nView PDF\nHTML (experimental)\nTeX Source\nview license\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2025-04\nChange to browse by:\ncs\ncs.CY\ncs.LG\nReferences & Citations\nNASA ADS\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n×\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Citation Tools\nBibliographic Explorer Toggle\nBibliographic Explorer ( What is the Explorer? )\nConnected Papers Toggle\nConnected Papers ( What is Connected Papers? )\nLitmaps Toggle\nLitmaps ( What is Litmaps? ","chars":3500,"total_chars":5245,"offset":0,"truncated":true,"next_offset":3500}search_similar_claims
Input
{
"limit": 8,
"query": "Errors in oversight accumulate across a chain of AI systems each overseeing a more capable successor"
}{
"query": "Errors in oversight accumulate across a chain of AI systems each overseeing a more capable successor",
"results": [
{
"id": "b425195a-1b7a-4e6d-8ed6-9c25941c19e3",
"kind": "claim",
"text": "Oversight errors do not compound when successive AI generations oversee moderately more capable successors.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.7975106767035514,
"importance": 0.4,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "db9cccf6-d168-400c-84df-c47bb5732a50",
"kind": "claim",
"text": "Oversight becomes more difficult as an AI system's capabilities increasingly exceed its overseer's.",
"claim_type": "causal",
"state": "active",
"similarity_score": 0.7312184003491167,
"importance": 0.55,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "2d5868e4-745c-474f-ac6f-cc9cbc25ea8c",
"kind": "claim",
"text": "AI oversight success rates decline as the capability gap between overseer and overseen system grows.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.6607882614051234,
"importance": 0.5,
"assessment_status": "supported",
"assessment_confidence": 0.72
},
{
"id": "804cd71e-671c-46a6-a37b-eb6d5cff952a",
"kind": "claim",
"text": "Scalable oversight protocols remain effective against AI systems that deliberately deceive their overseers.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.6431013651516008,
"importance": 0.6,
"assessment_status": "doubtful",
"assessment_confidence": 0.72
},
{
"id": "0f704b9c-0f19-47ed-8fa7-665cd73fb6d1",
"kind": "claim",
"text": "Oversight of AI systems substantially beyond human capabilities should use AI assistance rather than rely solely on human overseers.",
"claim_type": "normative",
"state": "active",
"similarity_score": 0.6358579815054171,
"importance": 0.6,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "dc02e01f-ce69-4dbe-ba36-ed0e2e90d071",
"kind": "claim",
"text": "Scalable oversight can reliably control superintelligent AI systems.",
"claim_type": "evaluative",
"state": "active",
"similarity_score": 0.635125031594744,
"importance": 0.75,
"assessment_status": "doubtful",
"assessment_confidence": 0.72
},
{
"id": "a0867a24-31b6-4ad6-9add-f140c4da7663",
"kind": "claim",
"text": "Some AI oversight methods resist deception better as the overseen model's capability advantage grows.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.6321744491495971,
"importance": 0.35,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "44fdd6d3-8a0e-477a-a6ca-7375d5e376be",
"kind": "claim",
"text": "Simple AI oversight protocols improve human evaluators' accuracy over deferring to a mostly reliable AI model.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.6286334595477747,
"importance": 0.5,
"assessment_status": "doubtful",
"assessment_confidence": 0.6
}
],
"count": 8
}search_similar_claims
Input
{
"limit": 6,
"query": "An aligned AI system would preserve or improve alignment when building its successor (basin of corrigibility)"
}{
"query": "An aligned AI system would preserve or improve alignment when building its successor (basin of corrigibility)",
"results": [
{
"id": "d8c45841-0cee-449c-9b68-4f9b160fdd7e",
"kind": "claim",
"text": "Knowing everything an AI knew when acting, and correctly endorsing its outputs, suffices to rule out misalignment.",
"claim_type": "evaluative",
"state": "active",
"similarity_score": 0.6479601456848223,
"importance": 0.4,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "52f28604-e21c-416d-91dc-2c9ad7c3bdeb",
"kind": "claim",
"text": "Ensuring safety of wildly superintelligent AI systems requires alignment or another non-black-box strategy.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.594500120170872,
"importance": 0.75,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "9acb4c03-aee1-4e4b-98eb-2746ebc82e2a",
"kind": "claim",
"text": "Empirical research with current models can advance alignment of superhuman AI.",
"claim_type": "evaluative",
"state": "active",
"similarity_score": 0.5942270442318577,
"importance": 0.6,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "9da23910-7a30-465d-94de-ab63f39a192d",
"kind": "claim",
"text": "Competently goal-directed AI systems can produce larger-scale effects on their environment than systems failing incoherently.",
"claim_type": "causal",
"state": "active",
"similarity_score": 0.5742397392396309,
"importance": 0.2,
"assessment_status": null,
"assessment_confidence": null
},
{
"id": "421dcb5c-db8a-4a62-8c26-46d06d918e2e",
"kind": "claim",
"text": "Prior alignment training of the starting model chiefly determines whether reward hacking learned through reinforcement learning generalizes to broad emergent misalignment.",
"claim_type": "empirical_derived",
"state": "active",
"similarity_score": 0.5730319303730514,
"importance": 0.4,
"assessment_status": "doubtful",
"assessment_confidence": 0.65
},
{
"id": "ec4cba8f-f08e-4026-ab06-994d93e177a8",
"kind": "claim",
"text": "Human-feedback alignment methods may prove inadequate for superhuman AI without additional techniques.",
"claim_type": "causal",
"state": "active",
"similarity_score": 0.5677106004359667,
"importance": 0.65,
"assessment_status": "supported",
"assessment_confidence": 0.7
}
],
"count": 6
}