Weak research judgment in AI systems will slow automation of the remaining AI R&D tasks below current-trend extrapolations.
Reassessed
Consistency flag (stale_premise) and mandate-plan reassess: the residual-task-composition premise is now assessed (supported, conf 0.5, credence 0.6) with notable residual long-horizon engineering. Replaced the stale "unjudged" wording in the assessment and the bottleneck argument's evaluation; added that the engineering share of the residual moves at trend rate, narrowing the bottleneck's reach. Credence 0.60 -> 0.55; status inconclusive unchanged. Change too small to warrant notifying dependents.
Reassessed: still Inconclusive
verdict confidence 0.65 · credence 0.55
Structured and assessed
Reworded to state the baseline (current-trend extrapolations). Built two arguments: judgment-bottleneck (for; new subclaim on residual task composition, plus existing failure and human-judgment claims) and fast-improving judgment (against; existing next-steps claim). Linked eventual-research-taste claim as related. Assessed inconclusive, credence 0.6, reconciling narrow-judgment improvement with open-ended judgment failures. No dependents to notify.
Assessed Inconclusive
verdict confidence 0.65 · credence 0.60
The claim is a forecast: that AI systems' weak research judgment (choosing goals and next steps, redesigning after flaws, backtracking, interpreting results) will hold back automation of the AI R&D work not yet automated, so that it proceeds more slowly than simple extrapolations of current trends suggest. Those extrapolations, built mostly on company-reported metrics, put most AI R&D in AI hands by 2027 or 2028. The case for it is substantial. By late 2026 AI writes most code and does several agent-workdays per human workday at the leading labs, yet humans still supply most research judgment, and Anthropic itself reports large gaps in its models choosing goals. The residual tasks appear concentrated in judgment. Independent "shadow evaluations" in which agents tackled the question of an unpublished paper found recurring judgment failures: underpowered results presented as findings, caveats added instead of redesign, early abandonment of ambitious directions; the original authors rejected all four AI-written papers. On open-ended reinvention tasks agents recover a small fraction of a published innovation's gain. When fast-moving components are automated, the slowest remaining component tends to govern overall pace. The case against is that measured judgment is improving quickly: Anthropic's internal next-step test rose from 51% to 64% preference over researchers between two model generations, and the external TasteVal benchmark shows experiment-selection efficiency improving on a steepening trend, with the best model above an expert baseline. The two bodies of evidence fit together once one notices what each measures. The improving measures test narrow judgment inside a framed problem (the next debugging step, which experiment to run on a fixed task with a fixed coder). The failure reports test broad, open-ended judgment: what is worth pursuing, when a design is broken, what counts as a result. Narrow judgment is plainly improving; whether broad judgment follows at the same pace is not yet measured, and it is the broad kind the residual tasks mostly need. The claim therefore turns on an unobserved rate. The bottleneck logic and the open-ended failures make some slowing relative to naive extrapolation more likely than not, but steep gains on the narrow measures, the speed of recent model turnover, and the small, recent evidence base on open-ended research (a handful of runs) keep the question open. Repeated open-ended evaluations across successive model generations, and company reporting on the autonomy of judgment-heavy tasks specifically, would settle it.
Claim entered the graph