The argument from parity in research taste
The argument
Current AI systems have research taste comparable to top human AI researchers: on TasteVal, an October 2026 benchmark of experiment design and interpretation on a fixed problem, the best model exceeded the best of 24 recruited experts. The stages at which earlier evaluations found agents failing, such as choosing experiments, judging evidence and changing course, are exercises of research taste. Those evaluations tested models released earlier in the year. So the frontier agents of late 2026 do not struggle with those stages, and the earlier failures no longer describe the frontier.
Does it work?
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 doubtful
Granting its premises, the conclusion would probably follow, since an agent with the research taste of a top researcher would not be expected to make the errors of judgment the earlier evaluations recorded. The argument stands or falls with parity in research taste, which the evidence currently weighs against. The benchmark it cites measures only experiment design and interpretation on a fixed, scored problem, with a separate agent doing the coding, against recruited experts. Measures of choosing and judging research directions still show models well below experienced researchers. It remains the most serious consideration against the claim, because it concerns models newer than those tested in open-ended evaluations.
Premises
- DoubtfulThe evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.constitution →Current AI systems have research taste comparable to top human AI researchers.
Where it is made
Passages where sources make, criticize or mention this argument. They record where it appears, not how strong it is, and none has precedence over another.
TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. [...] The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost.
Oliver Jaffe and Dane Sherburn · 2026-10-05
Earlier ratings
- Nov 13, 2026likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
Granting its premises, the conclusion would probably follow: an agent with the research taste of a top researcher would not be expected to make the errors of judgment the earlier evaluations recorded. The argument stands or falls with parity in research taste, which the evidence currently weighs against. The benchmark it cites measures only experiment design and interpretation on a fixed, scored problem with a separate agent doing the coding, against recruited experts; measures of choosing and judging research directions still show models well below experienced researchers. It remains the most serious consideration against the claim, because it concerns models newer than those tested in open-ended evaluations.