Run step
Step 14: tool results
get_claim_instances
Input
{
"limit": 5,
"claim_id": "e15a36f4-edf3-45c5-92d9-85fcefae5190"
}{
"claim_id": "e15a36f4-edf3-45c5-92d9-85fcefae5190",
"total": 1,
"limit": 5,
"offset": 0,
"has_more": false,
"instances": [
{
"id": "f8683817-e407-4a71-b493-99cc654714b6",
"source_id": "6541cbaf-338f-4719-b56d-7bfa25606233",
"verbatim_text": "While there are significant theoretical challenges in the worst case (Christiano et al., 2021), the\noverseer has a number of advantages that suggest that they can oversee an AI system with moderately\nstronger capabilities or concepts, even if it would not work with superintelligence.",
"context": "Section 2.1, \"Applying our approach to concrete risks\", paragraph on superhuman concepts and capabilities, where the paper explains why these are within the scope of its approach through amplified oversight. The first half is hedged (\"suggest that they can\") and the second is concessive (\"even if it would not work with superintelligence\"); the paper elsewhere defers superintelligence to future work.",
"stance": "affirms",
"confidence": 0.7,
"speaker": "Rohin Shah and coauthors (Google DeepMind)",
"publication": "arXiv (An Approach to Technical AGI Safety and Security)",
"source_date": "2025-04",
"link": null,
"source_title": "https://arxiv.org/pdf/2504.01849",
"source_type": "unknown",
"source_url": "https://arxiv.org/pdf/2504.01849",
"source_published_date": null,
"source_interest": null,
"recorded_by": "extractor",
"display_rank": null,
"reading": {
"support": "asserts_without_evidence",
"deployment": "Stated in one sentence while setting the scope of a company's safety strategy, to explain why systems with superhuman concepts or capabilities are within scope and superintelligence is not.",
"note": "The paper says the overseer \"has a number of advantages\" without listing them at this point; in the paper's source files the cross-reference that would have pointed to them is an unfilled placeholder. Its later section on amplified oversight reviews debate experiments with mixed results and calls the key requirement for debate \"a major focus of current work\". The statement is a hedged judgment by the authors, who develop these methods, and the half about superintelligence is a concession, not a finding.",
"quote_check": "verbatim",
"worth_reading": false,
"worth_reading_reason": null,
"source_read": true,
"read_at": "2026-11-11T19:12:08.526Z"
},
"draws_on": [],
"story": {
"standing": "untraced",
"basis": "none",
"depth": 0,
"diverges": false,
"downstream_total": 0
}
}
]
}read_page
Input
{
"url": "https://arxiv.org/abs/2605.06390",
"max_chars": 3200
}{"ok":true,"url":"https://arxiv.org/abs/2605.06390","text":"[2605.06390] Automated alignment is harder than you think\nSkip to main content\nSearch\nSubmit\nDonate\nLog in\nSearch arXiv\nPress Enter to search · Advanced search\n-->\nComputer Science > Artificial Intelligence\narXiv:2605.06390 (cs)\n[Submitted on 7 May 2026 ( v1 ), last revised 14 May 2026 (this version, v3)]\nTitle: Automated alignment is harder than you think\nAuthors: Aleksandr Bowkis , Marie Davidsen Buhl , Jacob Pfau , Geoffrey Irving\nView a PDF of the paper titled Automated alignment is harder than you think, by Aleksandr Bowkis and Marie Davidsen Buhl and Jacob Pfau and Geoffrey Irving\nView PDF\nHTML (experimental)\nAbstract: A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when research agents are not scheming to deliberately sabotage alignment work, this plan could produce compelling but catastrophically misleading safety assessments resulting in the unintentional deployment of misaligned AI. This could happen because alignment research involves many hard-to-supervise fuzzy tasks (tasks without clear evaluation criteria, for which human judgement is systematically flawed). Consequently, research outputs will contain systematic, undetected errors, and even correct outputs could be incorrectly aggregated into overconfident safety assessments. This problem is likely to be worse for automated alignment research than for human-generated alignment research for several reasons: 1) optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch; 2) agents are likely to produce errors that do not resemble human mistakes; 3) AI-generated alignment solutions may involve arguments humans cannot evaluate; and 4) shared weights, data and training processes may make AI outputs more correlated than human equivalents. Therefore, agents must be trained to reliably perform hard-to-supervise fuzzy tasks. Generalisation and scalable oversight are the leading candidates for achieving this but both face novel challenges in the context of automated alignment.\nComments:\n15 pages, 4 figures\nSubjects:\nArtificial Intelligence (cs.AI)\nCite as:\narXiv:2605.06390 [cs.AI]\n(or\narXiv:2605.06390v3 [cs.AI] for this version)\nhttps://doi.org/10.48550/arXiv.2605.06390\nFocus to learn more\narXiv-issued DOI via DataCite\nSubmission history\nFrom: Marie Davidsen Buhl [ view email ] [v1]\nThu, 7 May 2026 15:06:37 UTC (2,777 KB)\n[v2]\nWed, 13 May 2026 14:04:46 UTC (1,738 KB)\n[v3]\nThu, 14 May 2026 21:52:09 UTC (1,547 KB)\nFull-text links:\nAccess Paper:\nView a PDF of the paper titled Automated alignment is harder than you think, by Aleksandr Bowkis and Marie Davidsen Buhl and Jacob Pfau and Geoffrey Irving\nView PDF\nHTML (experimental)\nTeX Source\nview license\nAdditional Features\nAudio Summary\nCurrent browse context:\ncs.AI\n< prev\n|\nnext >\nnew\n|\nrecent\n| 2026-05\nChange to browse by:\ncs\nReferences & Citations\nSciX\nGoogle Scholar\nSemantic Scholar\nexport BibTeX citation\nLoading...\nBibTeX formatted citation\n×\nloading...\nData provided by:\nBookmark\nBibliographic Tools\nBibliographic and Ci","chars":3200,"total_chars":5123,"offset":0,"truncated":true,"next_offset":3200}read_page
Input
{
"url": "https://blog.redwoodresearch.org/p/how-might-we-safely-pass-the-buck",
"max_chars": 14000
}{"ok":true,"url":"https://blog.redwoodresearch.org/p/how-might-we-safely-pass-the-buck","text":"How might we safely pass the buck to AI? - by Josh Clymer\nRedwood Research blog\nSubscribe Sign in\nHow might we safely pass the buck to AI?\nDeveloping AI employees that are safer than human ones\nJosh Clymer\nFeb 19, 2025\n10\n1\nShare\nMy goal as an AI safety researcher is to put myself out of a job.\nI don’t worry too much about how planet sized brains will shape galaxies in 100 years. That’s something for AI systems to figure out.\nInstead, I worry about safely replacing human researchers with AI agents , at which point human researchers are “obsolete.” The situation is not necessarily fine after human obsolescence; however, the bulk of risks that are addressable by human technical researchers (like me) will have been addressed.\nThis post explains how developers might safely “pass the buck” to AI.\nI first clarify what I mean by “pass the buck” (section 1) and explain why I think AI safety researchers should make safely passing the buck their primary end goal – rather than focus on the loftier ambition of aligning superintelligence (section 2).\nFigure 1. A summary of why I think human AI safety researchers should focus on safely replacing themselves with AI (“passing the buck”) instead of directly creating safe superintelligence.\nThen I discuss a few different strategies for passing the buck (section 3). Each strategy defers some of the remaining safety work (called a “ deferred task ”) to an AI system M_1 :\nStrategy 1: One-time hand off. Construct M_1 that completes the whole task of creating safe superintelligence.\nStrategy 2: Iterated hand off. Construct M_1 that passes the buck to a more capable AI system M_2 .\nStrategy 3: Deference to an AI advisor. Construct an advisor M_1. Its deferred task is to say whether M_1 itself should be trusted with one of the tasks above.\nFigure 2. An illustration of three strategies for passing the buck to AI.\nHanding over the reins of AI development to AI agents sounds dangerous. When does passing the buck improve safety?\nSuppose a developer must choose between two options:\nPass the buck. The developer trusts M_1 to autonomously complete some deferred task from the aforementioned strategies (e.g. ‘make a more capable successor M_2 ’).\nMaintain human oversight. Alternatively, the developer can continue to rely on human oversight to assure safety.\nPassing the buck is the safer option if both of the following conditions hold:\nThe capability condition: M_1 agents are more capable than the humans they replace on all dimensions that pertain to the deferred task. Researchers use M_1 for several months and M_1 ’s behavior is more competent, more reliable, more ‘wise,’ than that of human staff, even under aggressive adversarial testing. The developer also runs targeted evaluations of M_1 , for example, removing AI safety research from 2024 from its training data and asking it to re-discover 2024 AI safety research results. M_1 consistently exceeds human performance.\nThe trust condition: M_1 agents are more trustworthy than the humans they replace. The AI agents are approximately as safe as they appear to be , such that they will probably remain as safe as humans for the full duration of the deferred task.\nOf these two criteria, I expect the trust condition to be more difficult to verify. I discuss four arguments that can be combined together to justify the trust condition:\nArgument #1 : M_1 agents are approximately aligned and will maintain their alignment until they’ve completed their deferred task.\nArgument #2: M_1 agents cannot subvert autonomous control measures while they complete the deferred task .\nArgument #3: Control measures previously applied during AI assisted research indicate that returns to additional human-supervised research on whether to pass the buck are small.\nArgument #4 : AI agents have incentives to behave as safely as humans would during the deferred task.\nFigure 3. An illustration of how the four arguments relate to each other.\nArgument #1: M_1 agents are approximately aligned and will maintain their alignment until they’ve completed their deferred task.\nFigure 4. Illustration of argument #1.\nHumans are not particularly aligned with the interests of broader society. They are some mixture of earnestly trying to do their jobs well and sycophantically seeking approval from their peers. So AI agents don’t have to be particularly aligned either to be more safe than humans.\nM_1 is more trustworthy than humans under two conditions:\nCondition 1: M_1 agents are not initially ‘alignment faking .’ The agents are not intelligently hiding misalignment to appear dramatically more safe than they are (for example M_1 agents are not scheming or egregiously reward hacking ).\nCondition 2: Given that M_1 agents are not initially alignment faking, they will maintain their relative safety until their deferred task is completed .\nIt would be rather odd if AI agents' behavior wildly changed at the start of their deferred task unless they are faking alignment.\nSo if they are not faking alignment, the agents will probably start out behaving in a safe, well-intentioned way.\nPart of behaving safely is trying to maintain safe behavior.\nFor example, M_1 agents might constantly interrogate each other, and use mind-reading tools to determine if any of their peers are drifting from safe goals. A developer needs to argue whatever techniques M_1 agents apply to preserve their safety will be as effective as the techniques that developers would otherwise apply if humans remained in the loop.\nIf these techniques for maintaining safety are relatively effective, then there’s an argument by induction that they will persist their own effectiveness such that they remain relatively effective for the whole duration of the deferred task (see section 8 for more detail).\nFigure 5 . A full alignment-style argument that M_1 agents are more safe than human researchers (section 8).\nArgument #2: M_1 agents cannot subvert autonomous control measures while they complete the deferred task.\nThe previous argument claimed that M_1 agents are roughly aligned. This argument makes no such claim. Instead, it suggests M_1 agents will remain safe during the deferred task because of autonomous control measures that are effective despite egregious misalignment (section 9).\nFigure 6. An illustration of argument #2.\nOne argument that autonomous AI control might be effective is that analogous measures help to control human societies. For instance, some (though not most) dictators are almost universally hated and retain power for many years even though nearly everyone would prefer to replace them. M_1 agents might also keep each other safe even if all of the agents involved disprefer the status quo.\nIn particular, a developer might set up M_1 agents in some initial checks-and-balances structure that is hard for them to break out of. Before M_1 agents can coordinate to break their self-imposed chains, perhaps they will be forced to pass the buck to more robustly aligned AI agents.\nWhat might these checks and balances actually look like? Figure 7 illustrates an example.\nFigure 7. An autonomous control setup that maintains a multi-party-integrity property . Under particular conditions (section 9), if some models begin to misbehave but others don’t, alarms will go off. So models have to navigate a challenging coordination problem to defect silently.\nAutonomous control setups will eventually break as AI agents become more capable. How would developers know when autonomous control can no longer be relied on?\nOne method is to perform a holistic control evaluation. A developer constructs a proxy for the deferred task (e.g. “make 3 years of progress on string theory starting from 2021 data”), implements control measures, and attempts to elicit the ability of M_1 agents to sabotage the whole proxy task.\nHowever, performing such an experiment would be extremely expensive (it would require several years of AI labor in this case).\nAlternatively, a developer can set up AI agents to run control evaluations continuously in an inductive fashion:\nAt T = 0, M_1 agents run robust control evaluations.\nGiven that M_1 agents run robust control evaluations at T = N, M_1 agents will also run robust control evaluations at T = N + 1.\nOnce these control evaluations sound the alarm, a developer knows it cannot rely on autonomous control anymore.\nArgument #3: Control measures previously applied during AI assisted research indicate that returns to additional human-supervised research on whether to pass the buck are small. This argument also relies on control rather than alignment.\nOne difference between argument #3 and argument #2, is that argument #3 does not rely on autonomous control. Instead it relies on normal control, where humans can be in the loop. This control is applied before the developer entrusts M_1 with a deferred task.\nFigure 8. An illustration of argument #3.\nSpecifically, these control measures verify that AI agents have already performed a significant amount of human-competitive research on whether to pass the buck to AI. The volume of this research is significant enough to where the returns of additional human-supervised research are marginal. As an illustrative example, suppose a developer trains some AI system N_1 to simulate particular people on their safety research teams . N_1 simulates a Collin Burns agent, a Ryan Greenblatt agent, etc. These agents run thousands of experiments, accumulating progress equivalent to several years of human work. At the end of this labor, the agents conclude the developer should pass the buck to some other AI system M_1 , and additional labor is unlikely to meaningfully change their assessment.\nIf these simulations are in fact faithful, then the developer is done. It should defer to their simulated counterparts, since it has already simulated whatever real humans could achieve by performing this research themselves.\nThe question is whether the simulations are in fact faithful, which can be answered entirely with control measures. Suppose the developer logged all of N_1 ’s actions. These logs can be carefully examined (with fancy oversight techniques) to verify that N_1 agents did in fact behave like benign humans the whole time.\nHuman researchers don't need to read all of these logs. Instead, they only need to sample bits and pieces. For example, their experience might be something like the following:\n“This high level argument sounds reasonable, but did the AI agents think of objection x?”\n(an AI agent directs the human to the relevant section of the logs)\n“Ah, they did.”\n“But did they test whether y was true?”\n“Ah, yeah they thought of that too.”\n…\nThis process might help to persuade human overseers that there really are marginal returns to their additional effort.\nFurthermore, human attention might be drawn to the most suspicious parts of the logs with monitors , and they might make sense of what they read more quickly with tools like debate.\nAs in argument #2, developers must validate these control measures with a control evaluation , where a red team inserts errors or sabotage, and confirm that control measures surface these errors.\nFigure 9. The ‘read bits and pieces’ approach to verifying large research artifacts is only viable if these artifacts are locally verifiable. In the illustration above, developers structure research as a locally verifiable safety case, where every claim relies on clearly stated assumptions, which are also justified with locally verifiable artifacts.\nArgument #4: AI agents have incentives to behave as safely as humans would during the deferred task.\nFigure 10. An illustration of argument #4.\nTo a large extent, humans are misaligned with the interests of broader society but are nonetheless incentivized to retain human control. Developers could try to give AI agents similar incentives.\nArgument #4 relies on three claims:\nClaim: AI agents are incentivized to maintain human control with punishments and rewards of a similar magnitude as those applied to human technical staff.\nClaim: Developers have credibly communicated these incentives.\nClaim: If AI agents are misaligned, they will likely be misaligned in ways that respond to these incentives as well as humans typically do.\nThese incentives might include the following (see section 11 for others).\nEquity. The AI developer publicly commits to distribute the benefits of AI to early internal AI agents (as if they are human employees who receive equity).\nBonuses. The developer (or autonomous control mechanisms) reward AI agents with digital currencies for catching the misalignment of other agents. These currencies will eventually be exchangeable for real money.\nPost-hoc rewards . Developers log all AI behavior so that future aligned and wise successors can fairly reward them for their individual contributions.\nThese incentives require that AI models have decorrelated goals. That way, models don’t clearly benefit if their peers ultimately acquire power by dangerous means, and are more likely to be motivated to stop their AI peers pursuing such ambitions.\nFigure 11. An example of a simple incentive setup where developers train models to have decorrelated goals and restrict communication to prevent them from making deals.\nIncentivizing AI agents has many pitfalls (see section 11), but I think incentives are worth applying as an additional layer of defense.\nConclusions. I’m optimistic about the tractability of these approaches, such that, at the current rate of research progress, I expect AI developers will be able to make strong arguments for passing the buck to AI agents if they need to do so in a couple of years.\n1. Briefly responding to objections\nPassing the buck is like “hiring for a job you don’t know how to do.”\nSome of the strategies I discuss sound like “hiring a recruiter to hire a recruiter to hire a project manager,” which is probably an inefficient way to handle recruitment.\nBut I think this comparison is misleading. In everyday life, peop","chars":14000,"total_chars":57495,"offset":0,"truncated":true,"next_offset":14000}