Obtaining an amplified-oversight signal (AI-assisted human judgment, as in debate) for every action of very powerful AI is too costly in training and in deployment.

What the claim needs. "Too costly" is a threshold word with no threshold given. Two readings were assessed: (a) cost beyond any developer's means, (b) cost a developer could bear but would decline. The paper that asserts the claim (Shah and coauthors, "An Approach to Technical AGI Safety and Security", arxiv.org/pdf/2504.01849, Section 6.3) gives one reason, "Human oversight is, and will continue to be, expensive", and no figure. Its Section 6.1 defines the amplified-oversight signal as a human's feedback on one input-output pair, says the methods "generally maintain humans in the loop", and adds that some human judgment "can be imitated or delegated" and that the form of human involvement "is an open question". So the claim is an uncosted assumption in its source, and the reading of the signal is open in the source itself. That question is carried by the subclaim that amplified oversight methods require a human judgment for each signal↗︎, which is not yet assessed.

Deployment, human in the loop (own arithmetic). Volume: about 2.5 billion prompts a day to ChatGPT in July 2025 (OpenAI figure reported by Axios, techcrunch.com/2025/07/21/chatgpt-users-send-2-5-billion-prompts-a-day/); Google reported more than 1.3 quadrillion tokens a month in October 2025, about 4.3e13 a day, or roughly 4e10 actions a day if an action is about 1,000 tokens (the divisor is an assumption). Human time per judgment is assumed, not measured: 10 seconds to 30 minutes. At 2.5e9 actions a day: 10 s gives 6.9e6 person-hours a day (about 870,000 full-time judges), 12.7 to 51 billion dollars a year at 5 to 20 dollars an hour; 5 min gives 2.1e8 person-hours a day (26 million judges), about 1.5 trillion dollars a year at 20 dollars an hour; 30 min gives 1.25e9 person-hours a day (156 million judges), about 9 trillion dollars a year. The comparison to revenue uses a figure of roughly 10 to 20 billion dollars a year for the largest chatbot developer in 2025, which was not re-verified in this pass. The conclusion does not depend on it: the central cases exceed any plausible revenue by about two orders of magnitude. No published measurement of human time per debate judgment was located in this pass; the one large human-judge study (Khan and coauthors, arxiv.org/abs/2402.06782) reports over a thousand annotator hours in total.

Training, human in the loop (own arithmetic). 1e6 actions at 1 min and 20 dollars an hour is about 0.33 million dollars; 1e9 actions at 30 min and 100 dollars an hour is 50 billion dollars. The range is five orders of magnitude and its lower part is affordable beside training runs costing hundreds of millions or more. The claim's training half therefore depends on the number of judged actions and on the latency human judges add to a training loop, neither of which any source costs. Practice is indirect evidence: Christiano and coauthors (arxiv.org/abs/1706.03741) trained from human feedback "on less than 1% of our agent's interactions", and RLHF pipelines since have used learned reward models for the same reason.

Automated judge. No published per-action overhead for debate was found. The three-to-ten-times figure is a structural estimate from protocol shape (one to three rounds, two debater generations per round over a growing transcript, plus a judge) and should be read as an order of magnitude. Measured overheads exist for cheaper proxies: Constitutional Classifiers reported "a 23.7% inference overhead" on production traffic (arxiv.org/abs/2501.18837); the follow-up cascade with activation probes is reported at about 1% overhead, a roughly fortyfold reduction (arXiv 2601.04603, read here through secondary summaries only). The 2% human audit budget in AI-control experiments (arxiv.org/abs/2312.06942) is an experimental parameter, not a finding about affordability. Doubly-efficient debate (arxiv.org/abs/2311.14125) shows a constant number of human-judgment queries per debate suffices in theory, which bounds the human cost per signal without removing it.

Weighing. On the human reading, deployment is near certain (about 0.97) and training likely (about 0.8). On the delegated reading the claim holds only in the weaker sense (b), which is probable given revealed preferences but is partly a judgment about willingness to pay. The canonical wording glosses the signal as AI-assisted human judgment, so the human reading dominates; the credence of 0.82 reflects the conjunction over both phases and residual weight on the delegated reading.

Search. Queries covered the asserting paper, debate experiments, AI-control protocols (trusted monitoring, resampling), classifier overheads, and usage volumes. No source denying the claim was found; the search for denials was light, and no dedicated search for measured debate cost per judgment was made beyond these queries.

What would change this. A measured cost per amplified judgment (human minutes and model tokens) from debate experiments; action counts for reinforcement-learning phases of frontier training; or a demonstration that a model judge reproduces amplified human judgment closely enough to count as the signal itself.