Obtaining an amplified-oversight signal (AI-assisted human judgment, as in debate) for every action of very powerful AI is too costly in training and in deployment.
Structured and assessed
First pass. Read the asserting paper's Sections 6.1 and 6.3 in its arXiv TeX source: the claim is stated as an anticipation with one reason (human oversight is expensive) and no costing. Reworded the canonical form to gloss "amplified-oversight signal" as AI-assisted human judgment, since the verdict turns on that meaning and the term is a term of art. Corrected the single instance to carry the Section 6.3 passage that actually states the claim, with authors and date; recorded its reading (asserts without evidence) and a material source map. Structure: three named arguments (scarcity of human judgment, for; protocol compute overhead, for; delegated judging, against) and one new subclaim, the definitional crux of whether each amplified-oversight signal requires a human judgment (Matcher: new), shared by the first and third arguments. Linked the AI-control monitoring-evaluation claim as related rather than as a premise, since how well cheap schemes work does not bear on what the expensive one costs. Assessment: supported, confidence 0.75, credence 0.82, from own arithmetic on public usage volumes (2.5e9 prompts/day; 1.3e15 tokens/month) with assumed human time per judgment of 10 s to 30 min; deployment half near settled on the human reading, training half dependent on uncosted training action counts; automated-judge overhead is an unmeasured structural estimate (3-10x); measured overheads exist only for cheap classifiers (23.7%, later about 1%). Importance 0.3, contestation 0.15. Filed a request to compile measured per-judgment human time, per-debate token cost and training action counts, which would narrow the intervals and could weaken the training half. Not done: the constitutional-classifier follow-up was read only via secondary summaries; the revenue comparison figure was not re-verified; search for denials was light.
Reassessed: still Supported
verdict confidence 0.75 · credence 0.82
Assessed Supported
verdict confidence 0.75 · credence 0.82
The claim is well supported for deployment and plausible but uncosted for training. It is asserted in the literature as an anticipation, with no cost estimate behind it; the support recorded here comes from arithmetic on public usage figures, not from any published costing of oversight protocols. "Too costly" has no stated threshold. The claim is true in a strong sense if the cost exceeds what any developer could pay, and in a weaker sense if the cost is a multiple of inference that developers could pay but would not. Which sense applies depends on whether each amplified-oversight signal needs a human judgment. Where it does, the deployment half is close to settled. One chatbot alone received about 2.5 billion prompts a day in mid-2025. If a human spends ten seconds on each at 5 dollars an hour, the bill is about 13 billion dollars a year, roughly that developer's whole reported revenue; at five minutes and 20 dollars an hour it is about 1.5 trillion dollars a year and 26 million full-time judges. The plausible range spans three orders of magnitude and its floor is already prohibitive. Scarcity worsens as systems grow more capable, since action volume grows and the supply of qualified human attention does not. The training half is weaker. Judging one million to one billion training actions at one to thirty minutes each and 20 to 100 dollars an hour costs between about 0.3 million and 50 billion dollars. The low end is affordable. The claim holds for training only if very powerful systems are trained on something like a hundred million actions or more, or if waiting on human judges slows training unacceptably. Both are likely and neither has been costed. Existing practice points the same way: reward models were introduced because human labels on every sample did not scale, and the founding experiment gave feedback on under 1% of interactions. Where the human's part is delegated to a model, no measured per-action cost of a full debate protocol was found. A structural estimate is three to ten times the cost of producing the action, since each round needs two further generations by the strongest model plus a judge. That is a choice about willingness to pay, not an impossibility. Measured figures exist only for much cheaper proxies: a production classifier system ran at 23.7% inference overhead, and its developer then engineered that down to roughly 1% with a cascade. Developers working to remove a 24% overhead are unlikely to accept several hundred percent on every action. Cheaper proxies (sampled human review, distilled judges, tiered monitors, fixed audit budgets such as the 2% used in control experiments) do not make the claim false. They are the response to it. They show that some oversight of every action is affordable; what is given up is amplified quality on most actions, and whether enough survives is a separate question.
Updated claim instance
Instance dc877dfe-2345-4625-802a-57f07786210b (https://arxiv.org/pdf/2504.01849): was stance=affirms, confidence=0.6; set confidence=0.95, verbatimText="Human oversight is, and will continue to be, expensive. Even if we presume that techniques from Amplified Oversight (Section 6.1) make it possible to determine the acceptability of a given AI action, for very powerful future models, this will be too costly to do for all actions during training or (even more so) deployment.", segmentId=null, speaker="Rohin Shah and coauthors (Google DeepMind)", publication="arXiv (An Approach to Technical AGI Safety and Security)", sourceDate="2025-04", context="Opening of Section 6.3, \"Robust AI oversight for training and monitoring\", where the cost of amplified oversight motivates delegating oversight to cheaper AI judges and monitors. The extended abstract states the weaker form: \"we anticipate that Amplified Oversight signal will be costly to obtain.\"". The recorded passage was the weaker extended-abstract sentence ("will be costly to obtain"), which does not state the claim; the claim's own wording comes from the opening of Section 6.3, read in the paper's TeX source (06-addressing-misalignment/robust-training.tex). Replaced the passage with that statement, added the authors, venue and date, and raised confidence accordingly.
Claim entered the graph