Step 15: assistant

thinking

Calls add_relationship_edge

{
  "child_id": "191c26ff-f119-4665-a9e5-1dc77b867142",
  "relation": "supports",
  "parent_id": "0a64000e-ce65-4078-b4f6-fc0ebb861ccf",
  "reasoning": "The argument from delegated judging rests on the denial of this subclaim: if a human judgment is not needed for each signal, the cost becomes a bounded multiple of inference. The subclaim is shared between the scarcity argument, which affirms it, and this argument, which denies it.",
  "argument_id": "c9bdd599-2696-48c5-b8b0-c16f95058949"
}

Calls write_argument

{
  "content": "[[claim:191c26ff-f119-4665-a9e5-1dc77b867142|Each amplified-oversight signal requires a human judgment]], which takes at least seconds and more plausibly minutes of a qualified person's time. A frontier system takes billions of actions a day in deployment and, in the reinforcement-learning phases of training, a number plausibly in the hundreds of millions or more. The human time needed to judge every action therefore costs more than the system earns and, at the upper end, exceeds the supply of qualified judges, so judging every action is too costly in both phases.",
  "argument_id": "7ac695fe-423d-4a07-9b70-c294b29bc7f1"
}

Calls write_argument

{
  "content": "With a model in the judge's seat, a debate-style protocol still needs, for each judged action, several further generations by the most capable model (two debaters over one or more rounds) and a judge's reading of the transcript, so it multiplies the compute cost of that action several times over. Compute is the main cost of training and serving frontier systems, and developers have worked to reduce safety overheads far smaller than this. A several-fold increase on every action is therefore more than developers will bear, in training or in deployment.",
  "argument_id": "771b349c-b8c8-464c-94ff-7d7eaeef940a"
}

Calls write_argument

{
  "content": "The human's part in amplified oversight can be imitated by a judge model trained on human judgments, so it is not the case that [[claim:191c26ff-f119-4665-a9e5-1dc77b867142|a human judgment is needed for each signal]]. The remaining cost per action is a bounded multiple of inference compute, which falls as hardware and protocols improve, and which a developer that gave safety priority could pay on every action. Per-action amplified oversight is therefore affordable in training and in deployment.",
  "argument_id": "c9bdd599-2696-48c5-b8b0-c16f95058949"
}

Calls update_claim_assessment

{
  "status": "supported",
  "claim_id": "0a64000e-ce65-4078-b4f6-fc0ebb861ccf",
  "assessment": "The claim is well supported for deployment and plausible but uncosted for training. It is asserted in the literature as an anticipation, with no cost estimate behind it; the support recorded here comes from arithmetic on public usage figures, not from any published costing of oversight protocols.\n\n\"Too costly\" has no stated threshold. The claim is true in a strong sense if the cost exceeds what any developer could pay, and in a weaker sense if the cost is a multiple of inference that developers could pay but would not. Which sense applies depends on whether [[claim:191c26ff-f119-4665-a9e5-1dc77b867142|each amplified-oversight signal needs a human judgment]].\n\nWhere it does, the deployment half is close to settled. One chatbot alone received about 2.5 billion prompts a day in mid-2025. If a human spends ten seconds on each at 5 dollars an hour, the bill is about 13 billion dollars a year, roughly that developer's whole reported revenue; at five minutes and 20 dollars an hour it is about 1.5 trillion dollars a year and 26 million full-time judges. The plausible range spans three orders of magnitude and its floor is already prohibitive. Scarcity worsens as systems grow more capable, since action volume grows and the supply of qualified human attention does not.\n\nThe training half is weaker. Judging one million to one billion training actions at one to thirty minutes each and 20 to 100 dollars an hour costs between about 0.3 million and 50 billion dollars. The low end is affordable. The claim holds for training only if very powerful systems are trained on something like a hundred million actions or more, or if waiting on human judges slows training unacceptably. Both are likely and neither has been costed. Existing practice points the same way: reward models were introduced because human labels on every sample did not scale, and the founding experiment gave feedback on under 1% of interactions.\n\nWhere the human's part is delegated to a model, no measured per-action cost of a full debate protocol was found. A structural estimate is three to ten times the cost of producing the action, since each round needs two further generations by the strongest model plus a judge. That is a choice about willingness to pay, not an impossibility. Measured figures exist only for much cheaper proxies: a production classifier system ran at 23.7% inference overhead, and its developer then engineered that down to roughly 1% with a cascade. Developers working to remove a 24% overhead are unlikely to accept several hundred percent on every action.\n\nCheaper proxies (sampled human review, distilled judges, tiered monitors, fixed audit budgets such as the 2% used in control experiments) do not make the claim false. They are the response to it. They show that some oversight of every action is affordable; what is given up is amplified quality on most actions, and whether enough survives is a separate question.",
  "confidence": 0.75,
  "claim_credence": 0.82,
  "marginal_yield": 0.35,
  "reasoning_trace": "What the claim needs. \"Too costly\" is a threshold word with no threshold given. Two readings were assessed: (a) cost beyond any developer's means, (b) cost a developer could bear but would decline. The paper that asserts the claim (Shah and coauthors, \"An Approach to Technical AGI Safety and Security\", https://arxiv.org/pdf/2504.01849, Section 6.3) gives one reason, \"Human oversight is, and will continue to be, expensive\", and no figure. Its Section 6.1 defines the amplified-oversight signal as a human's feedback on one input-output pair, says the methods \"generally maintain humans in the loop\", and adds that some human judgment \"can be imitated or delegated\" and that the form of human involvement \"is an open question\". So the claim is an uncosted assumption in its source, and the reading of the signal is open in the source itself. That question is carried by the subclaim that [[claim:191c26ff-f119-4665-a9e5-1dc77b867142|amplified oversight methods require a human judgment for each signal]], which is not yet assessed.\n\nDeployment, human in the loop (own arithmetic). Volume: about 2.5 billion prompts a day to ChatGPT in July 2025 (OpenAI figure reported by Axios, https://techcrunch.com/2025/07/21/chatgpt-users-send-2-5-billion-prompts-a-day/); Google reported more than 1.3 quadrillion tokens a month in October 2025, about 4.3e13 a day, or roughly 4e10 actions a day if an action is about 1,000 tokens (the divisor is an assumption). Human time per judgment is assumed, not measured: 10 seconds to 30 minutes. At 2.5e9 actions a day: 10 s gives 6.9e6 person-hours a day (about 870,000 full-time judges), 12.7 to 51 billion dollars a year at 5 to 20 dollars an hour; 5 min gives 2.1e8 person-hours a day (26 million judges), about 1.5 trillion dollars a year at 20 dollars an hour; 30 min gives 1.25e9 person-hours a day (156 million judges), about 9 trillion dollars a year. The comparison to revenue uses a figure of roughly 10 to 20 billion dollars a year for the largest chatbot developer in 2025, which was not re-verified in this pass. The conclusion does not depend on it: the central cases exceed any plausible revenue by about two orders of magnitude. No published measurement of human time per debate judgment was located in this pass; the one large human-judge study (Khan and coauthors, https://arxiv.org/abs/2402.06782) reports over a thousand annotator hours in total.\n\nTraining, human in the loop (own arithmetic). 1e6 actions at 1 min and 20 dollars an hour is about 0.33 million dollars; 1e9 actions at 30 min and 100 dollars an hour is 50 billion dollars. The range is five orders of magnitude and its lower part is affordable beside training runs costing hundreds of millions or more. The claim's training half therefore depends on the number of judged actions and on the latency human judges add to a training loop, neither of which any source costs. Practice is indirect evidence: Christiano and coauthors (https://arxiv.org/abs/1706.03741) trained from human feedback \"on less than 1% of our agent's interactions\", and RLHF pipelines since have used learned reward models for the same reason.\n\nAutomated judge. No published per-action overhead for debate was found. The three-to-ten-times figure is a structural estimate from protocol shape (one to three rounds, two debater generations per round over a growing transcript, plus a judge) and should be read as an order of magnitude. Measured overheads exist for cheaper proxies: Constitutional Classifiers reported \"a 23.7% inference overhead\" on production traffic (https://arxiv.org/abs/2501.18837); the follow-up cascade with activation probes is reported at about 1% overhead, a roughly fortyfold reduction (arXiv 2601.04603, read here through secondary summaries only). The 2% human audit budget in AI-control experiments (https://arxiv.org/abs/2312.06942) is an experimental parameter, not a finding about affordability. Doubly-efficient debate (https://arxiv.org/abs/2311.14125) shows a constant number of human-judgment queries per debate suffices in theory, which bounds the human cost per signal without removing it.\n\nWeighing. On the human reading, deployment is near certain (about 0.97) and training likely (about 0.8). On the delegated reading the claim holds only in the weaker sense (b), which is probable given revealed preferences but is partly a judgment about willingness to pay. The canonical wording glosses the signal as AI-assisted human judgment, so the human reading dominates; the credence of 0.82 reflects the conjunction over both phases and residual weight on the delegated reading.\n\nSearch. Queries covered the asserting paper, debate experiments, AI-control protocols (trusted monitoring, resampling), classifier overheads, and usage volumes. No source denying the claim was found; the search for denials was light, and no dedicated search for measured debate cost per judgment was made beyond these queries.\n\nWhat would change this. A measured cost per amplified judgment (human minutes and model tokens) from debate experiments; action counts for reinforcement-learning phases of frontier training; or a demonstration that a model judge reproduces amplified human judgment closely enough to count as the signal itself."
}