Step 14: tool results

{
  "content": "For many tasks a candidate solution can be tested far more cheaply than it can be found: a proof can be checked step by step, a program run against a specification, a factual answer traced to its sources. Ordinary delegated work shows the same pattern, since a paper is easier to review than to write and a tool's usefulness is easier to judge than the tool is to build. The outputs of AI systems whose correctness and safety matter are of this kind, or can be put in a form that is, and a check that takes less effort can also be carried out by a less capable checker. So checking an AI system's output takes less capability than producing it.",
  "argument_id": "49ff2637-aef9-424a-b25d-d3e375edaf41"
}

{
  "content": "A checker need not find flaws unaided. Protocols can make capable systems expose or contest each other's work, so that the checker only has to judge a narrow disputed point or read reasoning laid out for it. [[claim:4199e7f6-20f8-46d0-ad09-446ad43e796f|Debate lets a less capable judge supervise more capable debaters]], [[claim:54994356-47c8-4a8f-8761-c9ce277a8456|scaffolding and access to an agent's reasoning let weaker models monitor stronger agents]], and [[claim:804cd71e-671c-46a6-a37b-eb6d5cff952a|these protocols keep working when the overseen system is trying to deceive]]. With such protocols available, checking an output for correctness and safety takes less capability than producing it.",
  "argument_id": "70044d62-9094-46d7-8fad-13154bcfc5f0"
}

{
  "content": "The cheap-check asymmetry is drawn from problems chosen because a solution comes with something easy to test. Certifying that an output is safe is a different kind of task: it means ruling out every harmful flaw, and no certificate of absence is supplied. A producer has only to construct one output with a flaw the checker does not see, and a producer that optimizes against the check will select exactly such flaws; [[claim:1ab6aea0-bea9-470b-a3b1-c9d2564fef77|in debate, arguments can be built whose flaw is too costly for an opponent to locate]]. Writing a program that halts is trivial while deciding whether a given program halts is not computable. So for safety, and for correctness wherever the producer may be working against the checker, checking does not take less capability than producing.",
  "argument_id": "7f28dfac-7e77-4c55-9444-25b099489474"
}

{
  "content": "Experiments in which one model hides a backdoor or a sabotage step and another model or a person looks for it find that the share caught depends mainly on how capable the checker is relative to the producer: [[claim:3c5e9fc9-3a05-44b4-bd1d-2785b377594d|a weaker monitor catches less of an agent's hidden misbehavior as the agent's advantage grows]], attacks refined against monitors cut the catch rate of even frontier monitors, and unaided human developers miss most sabotage by coding agents. These experiments are representative of checking an AI system's output for safety. If checking needed less capability than producing, detection would hold up as the checker fell behind; it does not, so the claim fails for safety.",
  "argument_id": "e64e68f2-1881-4ca0-b366-a5761341af0d"
}

{
  "url": "https://www.lesswrong.com/posts/2PDC69DDJuAx6GANa/verification-is-not-easier-than-generation-in-general",
  "title": "Verification Is Not Easier Than Generation In General (comment thread)",
  "stance": "advances",
  "context": "Wentworth's reply to Christiano under his own post, generalizing the halting example to adversarial settings.",
  "reading": "full",
  "speaker": "John Wentworth",
  "argument_id": "7f28dfac-7e77-4c55-9444-25b099489474",
  "source_date": "2022-12-06",
  "verbatim_text": "the halting example suggests that verification is likely to be harder than generation specifically when there is some (possibly implicit) adversary. What makes verification potentially hard is the part where we have to quantify over all possible inputs—the verifier must work for any input. [...] It’s only easier to review papers than to write them because reviewers do not actually need to catch all problems. [...] Likewise with fraud: it’s only easier to catch than to perpetrate if we’re not expected to catch all of it, or even most of it. It’s easier to write secure software than to verify that a piece of software is secure."
}

{
  "url": "https://www.lesswrong.com/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer",
  "title": "Where I agree and disagree with Eliezer",
  "stance": "advances",
  "context": "Reply to Yudkowsky's claim that humans could not verify plans proposed by AI systems. The passage leaves unstated that AI outputs are like ordinary R&D outputs in the relevant respect and that lower effort implies lower required capability.",
  "reading": "premise_unstated",
  "speaker": "Paul Christiano",
  "argument_id": "49ff2637-aef9-424a-b25d-d3e375edaf41",
  "source_date": "2022-06",
  "verbatim_text": "I think it doesn’t match well with pragmatic experience in R&D in almost any domain, where verification is much, much easier than generation in virtually every domain."
}