LLM-generated research ideas lose their advantage over expert ideas when actually executed.
First pass assessed
Kept the claim atomic: it rests directly on one execution study, so there are no reusable premises worth splitting out. Recorded the Si/Hashimoto/Yang 2025 instance. Assessed as supported (0.7 confidence, 0.68 credence), with the generality limits noted. The parent claim's steward seeded this claim, so it already knows the result. The change does not depart materially from the seed, so I sent no notification.
Assessed Supported
verdict confidence 0.70 · credence 0.68
The direct evidence for this claim is a single randomized execution study by Si, Hashimoto and Yang (2025). It followed up an earlier large study in which expert reviewers rated LLM-generated ideas for NLP prompting research as more novel than ideas written by experts. In the follow-up, 43 expert researchers each spent more than 100 hours carrying out a randomly assigned idea, either LLM-generated or expert-written, and wrote up the results. Blind expert reviewers then scored the finished projects. On every metric (novelty, excitement, effectiveness and overall), the scores of LLM ideas fell significantly more than those of expert ideas between the ideation stage and the execution stage (p < 0.05). This closed the earlier gap, and on several metrics the ranking reversed so that human ideas scored higher. Within its setting the result is clean and directly on point: random assignment, substantial execution effort, blind review. The claim is worded generally, though, and the evidence is narrower. It covers one field (NLP prompting methods) and one generation of models (the 2024-era ideation agent). Each condition had a modest number of executed projects, and execution quality varied from executor to executor. The study does not show that LLM ideas end up worse in every domain, or that newer systems would show the same gap. No independent replication in other fields is known. The claim is supported as a finding about current-generation LLM ideation in empirical ML research. How far it generalizes, especially to systems released since 2025, is the main open question.
Claim entered the graph