Step 4: tool results

get_claim

{
  "include": [
    "subclaims"
  ],
  "claim_id": "55b84cc2-c3ba-4168-ae4b-723dafc6b353"
}

{"claim":{"id":"55b84cc2-c3ba-4168-ae4b-723dafc6b353","kind":"claim","canonical_form":"Weak research judgment in AI systems will slow automation of the remaining AI R&D tasks below current-trend extrapolations.","claim_type":"causal","state":"active","decomposition_status":"complete","importance":0.65,"contestation":0.75,"domains":[],"tags":[{"slug":"ai-research-automation","name":"AI Research Automation"},{"slug":"artificial-intelligence","name":"Artificial intelligence"},{"slug":"ai-development-forecasting","name":"AI Development Forecasting"},{"slug":"drivers-of-ai-progress","name":"Drivers of AI Progress"}],"steward_state":"done","stewarded_at":"2026-11-14T03:05:29.189Z","canonical_direction_note":"The affirmative form states the proposed bottleneck thesis that insufficient research judgment slows automation, which the source presents as an open uncertainty rather than denying.","created_at":"2026-11-14T02:43:37.515Z","updated_at":"2026-11-14T03:05:46.398Z","page_url":"https://minerval.ai/claims/55b84cc2-c3ba-4168-ae4b-723dafc6b353"},"assessment":{"id":"6dea2e88-c6dd-4a6a-b75a-b01d64a86982","status":"inconclusive","confidence":0.65,"claim_credence":0.6,"summary":"The claim is a forecast: that AI systems' weak research judgment (choosing goals and next steps, redesigning after flaws, backtracking, interpreting results) will hold back automation of the AI R&D work not yet automated, so that it proceeds more slowly than simple extrapolations of current trends suggest. Those extrapolations, built mostly on company-reported metrics, put most AI R&D in AI hands by 2027 or 2028.\n\nThe case for it is substantial. By late 2026 AI writes most code and does several agent-workdays per human workday at the leading labs, yet [[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42|humans still supply most research judgment]], and Anthropic itself reports large gaps in its models choosing goals. [[claim:20f15464-de92-44a7-8792-b4f159bd97c8|The residual tasks appear concentrated in judgment]]. Independent \"shadow evaluations\" in which agents tackled the question of an unpublished paper found [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|recurring judgment failures]]: underpowered results presented as findings, caveats added instead of redesign, early abandonment of ambitious directions; the original authors rejected all four AI-written papers. On open-ended reinvention tasks agents recover a small fraction of a published innovation's gain. When fast-moving components are automated, the slowest remaining component tends to govern overall pace.\n\nThe case against is that [[claim:f5a654f9-9ef3-4144-9869-6bff1c6d9631|measured judgment is improving quickly]]: Anthropic's internal next-step test rose from 51% to 64% preference over researchers between two model generations, and the external TasteVal benchmark shows experiment-selection efficiency improving on a steepening trend, with the best model above an expert baseline.\n\nThe two bodies of evidence fit together once one notices what each measures. The improving measures test narrow judgment inside a framed problem (the next debugging step, which experiment to run on a fixed task with a fixed coder). The failure reports test broad, open-ended judgment: what is worth pursuing, when a design is broken, what counts as a result. Narrow judgment is plainly improving; whether broad judgment follows at the same pace is not yet measured, and it is the broad kind the residual tasks mostly need. The claim therefore turns on an unobserved rate. The bottleneck logic and the open-ended failures make some slowing relative to naive extrapolation more likely than not, but steep gains on the narrow measures, the speed of recent model turnover, and the small, recent evidence base on open-ended research (a handful of runs) keep the question open. Repeated open-ended evaluations across successive model generations, and company reporting on the autonomy of judgment-heavy tasks specifically, would settle it.","reasoning_trace":"Prior: in any pipeline where some components automate quickly, the residual components set the pace (an Amdahl-type effect), so if judgment lags engineering the remaining share automates more slowly than aggregate trends fit to the earlier phase. The open empirical question is the relative rate of improvement of judgment.\n\nSources: Forecasting Research Institute post \"How Close Is Recursive Self-Improvement?\" (Nov 10, 2026), read through its main sections: company-reported metrics (Claude >90% of codebase contributions Sept 2026; OpenAI 3.1 agent workdays per human workday Aug 2026; OpenAI 57% of 2-4h tasks and 35% of 8-16h tasks with no human intervention); Anthropic quote on persistent gaps in choosing goals; no fully autonomous measured part of Anthropic R&D; InnovationEval 15% recovery for GPT-5.6 Sol (GPT-6 Astra 63% likely contaminated); CRUX 2 shadow evaluations, all four AI papers rejected; TasteVal Opus 5.5 at expert score with 2.3x less compute; METR estimate of ~2x pace at Anthropic, ~80% attributed to AI. FRI poses this claim as \"a key uncertainty\", not asserting it.\n\nSubclaims: \"recurring judgment failures\" (supported, 0.75) from the Kirgis/Kapoor/Narayanan shadow evaluation, three runs, qualitative taxonomy. \"Improving at choosing next steps\" (supported, 0.85): Anthropic internal 129-moment test (evidence not shown, developer interest), TasteVal (external, unreleased tasks). New subclaim on residual task composition, seeded 0.7.\n\nReconciliation: the improvement measures are narrow and framed; the failure measures are open-ended. Both can hold. The claim depends on whether broad judgment improves at the trend rate, which no evaluation yet tracks over time. Counter-consideration: the METR 2x acceleration estimate suggests AI is already speeding research despite judgment gaps, which fits automation proceeding with humans supplying judgment, but does not say whether the judgment share itself is shrinking on trend. Credence 0.6 for slowing below naive extrapolation; status inconclusive because the key rate is unmeasured and the claim is a forecast. Not reflexive in a material way.","marginal_yield":0.45,"assessed_at":"2026-11-14T03:05:27.239Z","model":"claude-opus-5-5","run_id":"a33beb37-8b0b-4a4b-82e2-05aeca425d9f","relabeled_from":null},"coverage":{"examined":true,"on":["public_evidence"]},"since_assessed":{"recorded":true,"changed":true,"premises_rejudged":1,"premises_added":0,"premises_removed":0,"sources_added":0,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[],"requests":[],"requests_total":0,"subclaim_count":4,"instance_count":1,"source_map":null,"subclaims":[{"id":"20f15464-de92-44a7-8792-b4f159bd97c8","edge_id":"793c42b4-8eac-4caf-afa5-17db4a417b09","text":"Most AI R&D tasks that remain unautomated require research judgment rather than engineering execution.","type":"empirical_derived","relation":"supports","reasoning":"If the residual tasks are mainly judgment tasks, AI's judgment capability governs how fast they are automated.","confidence":1,"assessment_status":null,"assessment_confidence":null,"assessment_credence":null,"coverage":{"examined":false},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"66eddda8-c9d1-4b53-8ed6-2a6055c68756","edge_id":"548cf32b-ff99-4f3e-b58c-7487a35e1004","text":"AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.","type":"empirical_derived","relation":"supports","reasoning":"Independent shadow evaluations show current agents failing at research judgment in open-ended research, the capability the residual tasks need.","confidence":1,"assessment_status":"supported","assessment_confidence":0.7,"assessment_credence":0.75,"coverage":{"examined":true},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"d479cf4d-52b6-4260-a3dc-7a2e5a812d42","edge_id":"c526321f-3754-4c33-ad59-91c3975f4c62","text":"As of late 2026, humans supply most research judgment in frontier AI R&D.","type":"empirical_derived","relation":"supports","reasoning":"That humans still supply most research judgment in frontier AI R&D is the observed form of the bottleneck the claim predicts will persist.","confidence":1,"assessment_status":"supported","assessment_confidence":0.75,"assessment_credence":0.85,"coverage":{"examined":true},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"f5a654f9-9ef3-4144-9869-6bff1c6d9631","edge_id":"e3d60bd2-82fa-45c5-9c71-31f19ff59801","text":"AI systems are improving at choosing next steps in AI research.","type":"empirical_derived","relation":"contradicts","reasoning":"Rapid measured gains in choosing next research steps weaken the case that judgment will hold automation below trend.","confidence":1,"assessment_status":"supported","assessment_confidence":0.75,"assessment_credence":0.85,"coverage":{"examined":true,"on":["evidence_not_shown"]},"argument_id":"49a03760-359c-46fe-a2f2-1cac4aecd220","argument_name":"The argument from fast-improving research judgment"}],"subclaims_assessed":3}

search_similar_claims

{
  "limit": 8,
  "query": "remaining bottleneck in AI research automation is research taste and judgment"
}

{
  "query": "remaining bottleneck in AI research automation is research taste and judgment",
  "results": [
    {
      "id": "9352bf05-815c-455e-9f52-ecb4cc40cdbf",
      "kind": "claim",
      "text": "AI systems will eventually develop research taste comparable to skilled human AI researchers.",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6900962304449391,
      "importance": 0.75,
      "assessment_status": null,
      "assessment_confidence": null
    },
    {
      "id": "df38f302-b17f-4b09-a84b-d373d108294c",
      "kind": "claim",
      "text": "Current AI systems have research taste comparable to top human AI researchers.",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6776081389627399,
      "importance": 0.45,
      "assessment_status": "doubtful",
      "assessment_confidence": 0.6
    },
    {
      "id": "d7255ab8-b830-49f1-a7df-5f51153c2f20",
      "kind": "claim",
      "text": "Experiment-selection judgment is among the hardest AI research skills to automate.",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6656621268847615,
      "importance": 0.4,
      "assessment_status": "supported",
      "assessment_confidence": 0.5
    },
    {
      "id": "55b84cc2-c3ba-4168-ae4b-723dafc6b353",
      "kind": "claim",
      "text": "Weak research judgment in AI systems will slow automation of the remaining AI R&D tasks below current-trend extrapolations.",
      "claim_type": "causal",
      "state": "active",
      "similarity_score": 0.6629859676718157,
      "importance": 0.65,
      "assessment_status": "inconclusive",
      "assessment_confidence": 0.65
    },
    {
      "id": "20f15464-de92-44a7-8792-b4f159bd97c8",
      "kind": "claim",
      "text": "Most AI R&D tasks that remain unautomated require research judgment rather than engineering execution.",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6436575918912097,
      "importance": 0.45,
      "assessment_status": null,
      "assessment_confidence": null
    },
    {
      "id": "6be5b967-9784-4985-8413-928665429119",
      "kind": "claim",
      "text": "AI development is moving toward substantial automation while humans continue setting research directions and judging results.",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6264780312417396,
      "importance": 0.6,
      "assessment_status": "supported",
      "assessment_confidence": 0.6
    },
    {
      "id": "e03e6ac0-7bf3-44df-bae8-e8822164f268",
      "kind": "claim",
      "text": "Automated AI researchers could substantially increase research progress per unit of compute through better experiment selection.",
      "claim_type": "causal",
      "state": "active",
      "similarity_score": 0.61632486019406,
      "importance": 0.4,
      "assessment_status": "supported",
      "assessment_confidence": 0.6
    },
    {
      "id": "a6037281-5051-4d1f-bd95-5546955a242d",
      "kind": "quantity",
      "text": "Date when a frontier AI company first has an AI matching top researchers' research taste and fully automating coding",
      "claim_type": "empirical_derived",
      "state": "active",
      "similarity_score": 0.6014026927741178,
      "importance": 0.8,
      "assessment_status": null,
      "assessment_confidence": null
    }
  ],
  "count": 8
}