LLM-generated research ideas lose their advantage over expert ideas when actually executed.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The direct evidence for this claim is a single randomized execution study by Si, Hashimoto and Yang (2025). It followed up an earlier large study in which expert reviewers rated LLM-generated ideas for NLP prompting research as more novel than ideas written by experts. In the follow-up, 43 expert researchers each spent more than 100 hours carrying out a randomly assigned idea, either LLM-generated or expert-written, and wrote up the results. Blind expert reviewers then scored the finished projects. On every metric (novelty, excitement, effectiveness and overall), the scores of LLM ideas fell significantly more than those of expert ideas between the ideation stage and the execution stage (p < 0.05). This closed the earlier gap, and on several metrics the ranking reversed so that human ideas scored higher.
Within its setting the result is clean and directly on point: random assignment, substantial execution effort, blind review. The claim is worded generally, though, and the evidence is narrower. It covers one field (NLP prompting methods) and one generation of models (the 2024-era ideation agent). Each condition had a modest number of executed projects, and execution quality varied from executor to executor. The study does not show that LLM ideas end up worse in every domain, or that newer systems would show the same gap. No independent replication in other fields is known. The claim is supported as a finding about current-generation LLM ideation in empirical ML research. How far it generalizes, especially to systems released since 2025, is the main open question.
Full reasoning: the evidence and decisions behind this verdict
Primary source read (arXiv 2506.20803, abstract): the scores of LLM-generated ideas drop significantly more than expert ideas after execution on all four metrics (p<0.05), closing the ideation-stage gap, and on many metrics the ranking flips. This matches the claim's wording "lose their advantage" directly. Reasons the credence is not higher: (1) there is one study in one subfield, NLP prompting; (2) the ideas came from a 2024 LLM agent, and by late 2026 models are substantially stronger, so a general present-tense reading carries real uncertainty; (3) samples are modest and executor quality is a source of variance; (4) the "advantage" in question was itself mainly about rated novelty, a reviewer-judgment construct. I found no contrary execution-based study. Studies of automated AI-scientist pipelines generally report weak execution outcomes, which is loosely consistent with the claim. The preliminary seed (0.75) is close; I set it slightly lower because of the general wording and model drift. What would change this: an execution-stage replication with newer models or in another domain.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Its one source has been traced to an origin, or to one another.
Comparing the review scores of the same ideas before and after execution, the scores of the LLM-generated ideas decrease significantly more than expert-written ideas on all evaluation metrics (novelty, excitement, effectiveness, and overall; p < 0.05), closing the gap between LLM and human ideas observed at the ideation stage.
Execution study: 43 expert researchers each spent 100+ hours executing randomly assigned NLP ideas (LLM-generated or expert-written), with blind expert review of the resulting short papers.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="2e6a6348-a373-48b5-b7fa-f26a3c3342c5"><a href="https://minerval.ai/claims/2e6a6348-a373-48b5-b7fa-f26a3c3342c5">LLM-generated research ideas lose their advantage over expert ideas when actually executed.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/2e6a6348-a373-48b5-b7fa-f26a3c3342c5" title="Minerval: LLM-generated research ideas lose their advantage over expert ideas when actually execute…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/2e6a6348-a373-48b5-b7fa-f26a3c3342c5)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Stated views None yet. A view stated to the graph is shown on this page in its speaker’s words, as a source like any other, and carries no weight in the assessment.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Nov 5, 2026. Every judgment on this page is accompanied by a reasoning trace.