Step 14: tool results

{
  "relation": "supports",
  "parent_id": "0a64000e-ce65-4078-b4f6-fc0ebb861ccf",
  "reasoning": "Whether per-action amplified oversight is prohibitively costly or merely a multiple of inference cost turns on whether each signal consumes human attention. If it does, the scarcity argument carries the parent; if the human's part can be delegated, the parent rests on the weaker compute-overhead argument. The parent has that second line of support, so the relation is supports, not requires.",
  "seed_note": "The term's originators define the amplified-oversight signal as a human's feedback on one AI input-output pair, improved by AI assistance, and say the methods \"generally maintain humans in the loop\". The same text adds that some aspects of human judgment \"can be imitated or delegated\" and that the form of human involvement is an open question. Debate experiments routinely substitute model judges for humans, and doubly-efficient debate bounds the human's part at a constant number of queries per debate without removing it. So the strict per-signal requirement holds for the protocols as defined and is relaxed in most proposals for using them at scale.",
  "child_text": "Amplified oversight methods such as debate require a human judgment for each oversight signal they produce.",
  "claim_type": "definitional",
  "importance": 0.3,
  "argument_id": "7ac695fe-423d-4a07-9b70-c294b29bc7f1",
  "contestation": 0.35,
  "seed_credence": 0.55
}

{
  "status": "supported",
  "claim_id": "0a64000e-ce65-4078-b4f6-fc0ebb861ccf",
  "assessment": "The claim is well supported for deployment and plausible but uncosted for training. It is asserted in the literature as an anticipation, with no cost estimate behind it; the support recorded here comes from arithmetic on public usage figures, not from any published costing of oversight protocols.\n\n\"Too costly\" has no stated threshold. The claim is true in a strong sense if the cost exceeds what any developer could pay, and in a weaker sense if the cost is a multiple of inference that developers could pay but would not. Which sense applies depends on whether [[claim:PLACEHOLDER|each amplified-oversight signal needs a human judgment]].\n\nWhere it does, the deployment half is close to settled. One chatbot alone received about 2.5 billion prompts a day in mid-2025. If a human spends ten seconds on each at 5 dollars an hour, the bill is about 13 billion dollars a year, roughly that developer's whole reported revenue; at five minutes and 20 dollars an hour it is about 1.5 trillion dollars a year and 26 million full-time judges. The plausible range spans three orders of magnitude and its floor is already prohibitive. Scarcity worsens as systems grow more capable, since action volume grows and the supply of qualified human attention does not.\n\nThe training half is weaker. Judging one million to one billion training actions at one to thirty minutes each and 20 to 100 dollars an hour costs between about 0.3 million and 50 billion dollars. The low end is affordable. The claim holds for training only if very powerful systems are trained on something like a hundred million actions or more, or if waiting on human judges slows training unacceptably. Both are likely and neither has been costed. Existing practice points the same way: reward models were introduced because human labels on every sample did not scale, and the founding experiment gave feedback on under 1% of interactions.\n\nWhere the human's part is delegated to a model, no measured per-action cost of a full debate protocol was found. A structural estimate is three to ten times the cost of producing the action, since each round needs two further generations by the strongest model plus a judge. That is a choice about willingness to pay, not an impossibility. Measured figures exist only for much cheaper proxies: a production classifier system ran at 23.7% inference overhead, and its developer then engineered that down to roughly 1% with a cascade. Developers working to remove a 24% overhead are unlikely to accept several hundred percent on every action.\n\nCheaper proxies (sampled human review, distilled judges, tiered monitors, fixed audit budgets such as the 2% used in control experiments) do not make the claim false. They are the response to it. They show that some oversight of every action is affordable; what is given up is amplified quality on most actions, and whether enough survives is a separate question.",
  "confidence": 0.75,
  "claim_credence": 0.82,
  "marginal_yield": 0.35,
  "reasoning_trace": "What the claim needs. \"Too costly\" is a threshold word with no threshold given. Two readings were assessed: (a) cost beyond any developer's means, (b) cost a developer could bear but would decline. The paper that asserts the claim (Shah and coauthors, \"An Approach to Technical AGI Safety and Security\", https://arxiv.org/pdf/2504.01849, Section 6.3) gives one reason, \"Human oversight is, and will continue to be, expensive\", and no figure. Its Section 6.1 defines the amplified-oversight signal as a human's feedback on one input-output pair, says the methods \"generally maintain humans in the loop\", and adds that some human judgment \"can be imitated or delegated\" and that the form of human involvement \"is an open question\". So the claim is an uncosted assumption in its source, and the reading of the signal is open in the source itself.\n\nDeployment, human in the loop (own arithmetic). Volume: about 2.5 billion prompts a day to ChatGPT in July 2025 (OpenAI figure reported by Axios, https://techcrunch.com/2025/07/21/chatgpt-users-send-2-5-billion-prompts-a-day/); Google reported more than 1.3 quadrillion tokens a month in October 2025, about 4.3e13 a day, or roughly 4e10 actions a day if an action is about 1,000 tokens (the divisor is an assumption). Human time per judgment is assumed, not measured: 10 seconds to 30 minutes. At 2.5e9 actions a day: 10 s gives 6.9e6 person-hours a day (about 870,000 full-time judges), 12.7 to 51 billion dollars a year at 5 to 20 dollars an hour; 5 min gives 2.1e8 person-hours a day (26 million judges), about 1.5 trillion dollars a year at 20 dollars an hour; 30 min gives 1.25e9 person-hours a day (156 million judges), about 9 trillion dollars a year. The comparison to revenue uses a figure of roughly 10 to 20 billion dollars a year for the largest chatbot developer in 2025, which was not re-verified in this pass. The conclusion does not depend on it: the central cases exceed any plausible revenue by about two orders of magnitude. No published measurement of human time per debate judgment was located in this pass; the one large human-judge study (Khan and coauthors, https://arxiv.org/abs/2402.06782) reports over a thousand annotator hours in total.\n\nTraining, human in the loop (own arithmetic). 1e6 actions at 1 min and 20 dollars an hour is about 0.33 million dollars; 1e9 actions at 30 min and 100 dollars an hour is 50 billion dollars. The range is five orders of magnitude and its lower part is affordable beside training runs costing hundreds of millions or more. The claim's training half therefore depends on the number of judged actions and on the latency human judges add to a training loop, neither of which any source costs. Practice is indirect evidence: Christiano and coauthors (https://arxiv.org/abs/1706.03741) trained from human feedback \"on less than 1% of our agent's interactions\", and RLHF pipelines since have used learned reward models for the same reason.\n\nAutomated judge. No published per-action overhead for debate was found. The three-to-ten-times figure is a structural estimate from protocol shape (one to three rounds, two debater generations per round over a growing transcript, plus a judge) and should be read as an order of magnitude. Measured overheads exist for cheaper proxies: Constitutional Classifiers reported \"a 23.7% inference overhead\" on production traffic (https://arxiv.org/abs/2501.18837); the follow-up cascade with activation probes is reported at about 1% overhead, a roughly fortyfold reduction (arXiv 2601.04603, read here through secondary summaries only). The 2% human audit budget in AI-control experiments (https://arxiv.org/abs/2312.06942) is an experimental parameter, not a finding about affordability. Doubly-efficient debate (https://arxiv.org/abs/2311.14125) shows a constant number of human-judgment queries per debate suffices in theory, which bounds the human cost per signal without removing it.\n\nWeighing. On the human reading, deployment is near certain (about 0.97) and training likely (about 0.8). On the delegated reading the claim holds only in the weaker sense (b), which is probable given revealed preferences but is partly a judgment about willingness to pay. The canonical wording glosses the signal as AI-assisted human judgment, so the human reading dominates; the credence of 0.82 reflects the conjunction over both phases and residual weight on the delegated reading.\n\nSearch. Queries covered the asserting paper, debate experiments, AI-control protocols (trusted monitoring, resampling), classifier overheads, and usage volumes. No source denying the claim was found; the search for denials was light and no dedicated search for measured debate cost per judgment beyond these queries was made.\n\nWhat would change this. A measured cost per amplified judgment (human minutes and model tokens) from debate experiments; action counts for reinforcement-learning phases of frontier training; or a demonstration that a model judge reproduces amplified human judgment closely enough to count as the signal itself."
}

{
  "note": "The paper gives one reason, that human oversight is and will remain expensive, and no estimate of what per-action amplified oversight would cost. Its own definition leaves open how much of the human's part can be delegated. Sections 6.1 and 6.3 were read in the paper's source; the rest of the paper was not.",
  "support": "asserts_without_evidence",
  "deployment": "Stated as an anticipation at the opening of the section on robust training and monitoring, to motivate delegating oversight to cheaper AI judges and monitors.",
  "instance_id": "dc877dfe-2345-4625-802a-57f07786210b",
  "source_read": true,
  "worth_reading": false
}

{
  "summary": "The one recorded source, a 2025 Google DeepMind paper on technical AGI safety, states the claim as an anticipation and gives no cost estimate for it. The figures in the assessment are the graph's own arithmetic on public usage numbers, not a published costing.",
  "material": true
}

{
  "ask": "Compile measured costs of producing one amplified-oversight judgment from published debate and critique experiments: human judge minutes per judgment, and model tokens (or dollars) per debate relative to the tokens of the judged answer. Also compile any published counts of judged actions or episodes in reinforcement-learning phases of frontier model training.",
  "why": "The assessment that per-action amplified oversight is too costly rests on assumed ranges (10 seconds to 30 minutes of human time; three to ten times inference for an automated debate) because no measured figure was found. Measured human times near the low end and small training action counts would make the training half of the claim doubtful; a measured automated overhead near one to two times inference would weaken the claim on the delegated-judge reading; figures inside the assumed ranges would confirm it and narrow the intervals.",
  "claim_ids": [
    "0a64000e-ce65-4078-b4f6-fc0ebb861ccf"
  ],
  "who_could": "A researcher reading the appendices of four to six papers; no new experiments needed.",
  "budget_usd": 6,
  "satisfied_by": "A table of per-judgment human time and per-debate token cost drawn from the methods and appendices of the main debate studies (for example Khan et al. 2024, Kenton et al. 2024, Michael et al. 2023, Arnesen et al. 2024), each with its locator, plus any published training-volume figures, or a statement that the papers do not report them."
}