argumentA line of reasoning from premises to a conclusion. Like a claim, it belongs to no one who makes it.constitution →

The argument that decomposition breaks down on the tasks that matter

againstThis argument, if it holds, weighs against the claim.constitution →Iterated amplification can usefully train AI systems on complex real-world tasks for which no other training strategy is available.

The argument

Tasks with no human or algorithmic training signal are those too large for a person to perform or check. On such tasks a human's decomposition yields trees of subtasks too large to supervise, the subtasks depend on context that an isolated sub-agent does not see, and a large argument can hide a flaw that no participant can locate, so only arguments small relative to training computation can be trusted. Reliable human decomposition into context-limited subtasks therefore fails on exactly these tasks, the amplified system's answers are not a trustworthy training target, and a model trained to imitate them is not usefully trained.

Does it work?

Does it follow?: likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Do the premises hold?: 1 inconclusive

Granting its premises, the conclusion is probable: if decompositions of the relevant tasks are too large to supervise, depend on context sub-agents lack, and can hide flaws nobody finds, the amplified answers are a poor training target. It falls short of certain because the method only needs the amplified system to be somewhat better than the model alone. The weight rests on the obfuscated arguments limit, unassessed and of unsettled reach beyond debate, and on premises established for unaided humans under tight limits but open for amplifiers built from capable language models.

Premises

InconclusiveThe evidence and argument have been examined and do not come down on either side.constitution →Obfuscated arguments limit debate and iterated amplification to verifying arguments small relative to training computation.

Where it is made

Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.

Progress Update October 2019makes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
After running a few experiments, we realized that breaking down tasks of any interesting complexity using factored generation requires extremely large trees of work.
Ought · Ought · 2019-10-28
Debate update: Obfuscated arguments problemmakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
However, we discovered that a dishonest debater can often create arguments that have a fatal error, but where it is very hard to locate the error. We don’t have a fix for this “obfuscated argument” problem, and believe it might be an important quantitative limitation for both IDA and Debate.
Beth Barnes and Paul Christiano · AI Alignment Forum · 2020-12-23
Recursively Summarizing Books with Human Feedbackmakes itleaves a premise unstatedThe passage relies on a premise the inference needs without saying it.constitution →
Task decomposition assumes that separate parts of the task can be completed independently. However, this may not be true for summarizing books.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano · arXiv · 2021-09

Earlier ratings

Nov 11, 2026likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Granting its premises, the conclusion is probable: if decompositions of the relevant tasks are too large to supervise, depend on context sub-agents lack, and can hide flaws nobody can find, the amplified answers are a poor training target. It falls short of certain because the method only needs the amplified system to be somewhat better than the model alone, and partial or noisy decomposition might still provide that. The weight rests on the obfuscated arguments limit, which is unassessed and whose reach beyond debate is unsettled, and on the two premises stated in the prose. The evidence for those comes from experiments with human participants in 2018 and 2019 and from one book summarization study, so they are established for unaided humans under tight limits and remain open for amplifiers built from capable language models with large context.