Step 4: tool results

get_claim

{
  "include": [
    "subclaims"
  ],
  "claim_id": "3f5f6119-5e2a-456c-930e-ecba1fc88447"
}

{"claim":{"id":"3f5f6119-5e2a-456c-930e-ecba1fc88447","kind":"claim","canonical_form":"In evaluated long-horizon AI R&D tasks, frontier agents' strongest solutions mostly adapt or combine established techniques; methodological novelty is rare.","claim_type":"empirical_derived","state":"active","decomposition_status":"complete","importance":0.5,"contestation":0.45,"domains":[],"tags":[{"slug":"ai-research-automation","name":"AI Research Automation"},{"slug":"ai-capability-evaluation","name":"AI Capability Evaluation"},{"slug":"ai-agents","name":"AI Agents"}],"steward_state":"done","stewarded_at":"2026-11-14T03:04:29.079Z","canonical_direction_note":"This affirmative form states the reported pattern that agents' strongest task solutions generally reuse established methods rather than introducing novel methodology.","created_at":"2026-11-14T02:37:11.165Z","updated_at":"2026-11-14T03:04:39.659Z","page_url":"https://minerval.ai/claims/3f5f6119-5e2a-456c-930e-ecba1fc88447"},"assessment":{"id":"00f6805b-ab31-4392-91ab-5a3c42be47dd","status":"supported","confidence":0.8,"claim_credence":0.88,"summary":"The claim describes what published evaluations of AI agents on long-horizon AI research tasks have found, and two independent evaluations from 2026 bear it out.\n\nA systematic evaluation by Li and colleagues (August 2026) ran seven frontier models, among them GPT-5.5, Gemini-3.1-Pro, DeepSeek-V4-Pro and Kimi-K2.7-Code, mostly in a shared agent harness, on 36 long-horizon tasks spanning areas such as CUDA kernels and model development. Agents often reached competitive scores, but when LLM judges reviewed the best solution from each model on each task, only 3 of 252 counted as novel approaches. The authors conclude that agents behave \"more like engineering optimizers than fully autonomous researchers.\" Epoch AI's InnovationEval (October 2026) took a different approach. It asked agents to invent a post-training method matching a recent human innovation, on-policy self-distillation, without having seen it. The uncontaminated models, Claude Fable 5 and GPT-5.6 Sol, spent thousands of dollars of GPU time and made little progress. The best discovery, a self-imitation loss close to existing ideas, fell short of even a \"moderately interesting\" result. Successor models such as GPT-6 Astra had memorized the target paper, and their gains came mostly from reproducing it or from hyperparameter tuning.\n\nThe two studies differ in authors, models, tasks and how they judged novelty, and they agree. That makes the claim, as scoped to evaluated tasks, well supported. Some caveats limit how far it generalizes. Novelty was judged by LLM reviewers in one study and by a single author's expert comparison in the other. Benchmark tasks with fixed metrics and short budgets may favour the tuning of known methods. InnovationEval rests on one task and a few runs. Memorization can inflate scores without adding novelty, which actually strengthens the claim's reading. Neither study tests systems the leading labs held internally in late 2026, or agents given longer horizons or human guidance. The claim says nothing about those, and its truth does not carry over to them.","reasoning_trace":"Sources read directly: the abstract and introduction of arXiv 2608.13417 (https://arxiv.org/html/2608.13417), and the methodology, results and appendix sections of https://epoch.ai/publications/innovationeval.\n\nLi et al.: 7 models, 36 tasks, about $100k of inference, best@3 and avg@3 scoring, rule-based process metrics, and a complementary LLM-judge novelty review. Only 3 of 252 best-seed solutions (36 tasks × 7 models) qualified as novel. The paper's own judgment of novelty depends on the judges' protocol, which this pass did not inspect in detail. The tasks are optimization tasks with defined metrics, which plausibly reward adapting known techniques. That is a reason to limit the claim to such tasks, not to doubt it.\n\nEpoch InnovationEval: one task (rediscovering something matching SDPO, with a Qwen3-8B base, beating GRPO), a long prompt that explicitly demanded novelty, and a few runs each. Claude Fable 5 and GPT-5.6 Sol showed no memorization. GPT-6 Astra and Claude Fable 5.1 did show memorization; Astra's score came mostly from partially reimplementing SDPO, and Fable 5.1's 40% came mostly from hyperparameter tuning. In an ablation where Fable 5 was given the paper, it got close to the reference but stayed below it. The author judged no discovery to clear the \"moderately interesting\" tier.\n\nOn independence: different groups (Li et al. appear to be an industry and academic team; Epoch is an independent research nonprofit), different models, tasks and novelty criteria. Neither has an obvious stake in finding low novelty. Epoch's stated aim of tracking progress toward automated AI R&D cuts neither way.\n\nCredence of 0.88 that the claim, as scoped to evaluated tasks, holds. The residual doubt covers how robust the novelty judgments are and whether other evaluations (e.g. RE-Bench, ResearchGym, autoresearch-style results) show more novelty. This pass did not search them systematically. What would change the verdict: an evaluation showing frequent judged-novel methods from agents, or a re-review of the Li et al. novelty protocol finding that it was too strict.","marginal_yield":0.25,"assessed_at":"2026-11-14T03:04:29.700Z","model":"claude-opus-5-5","run_id":"7a65f508-9586-473f-af6d-d691ac79ae7d","relabeled_from":null},"coverage":{"examined":true,"on":["public_evidence"]},"since_assessed":{"recorded":true,"changed":false,"premises_rejudged":0,"premises_added":0,"premises_removed":0,"sources_added":0,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[],"requests":[],"requests_total":0,"subclaim_count":0,"instance_count":2,"source_map":null,"subclaims":[],"subclaims_assessed":0}