Step 2: tool results

get_claim

{
  "include": [
    "arguments",
    "subclaims"
  ],
  "claim_id": "55b84cc2-c3ba-4168-ae4b-723dafc6b353"
}

{"claim":{"id":"55b84cc2-c3ba-4168-ae4b-723dafc6b353","kind":"claim","canonical_form":"Weak research judgment in AI systems will slow automation of the remaining AI R&D tasks below current-trend extrapolations.","claim_type":"causal","state":"active","decomposition_status":"complete","importance":0.65,"contestation":0.75,"domains":[],"tags":[{"slug":"ai-research-automation","name":"AI Research Automation"},{"slug":"artificial-intelligence","name":"Artificial intelligence"},{"slug":"ai-development-forecasting","name":"AI Development Forecasting"},{"slug":"drivers-of-ai-progress","name":"Drivers of AI Progress"}],"steward_state":"running","stewarded_at":"2026-11-15T07:12:36.213Z","canonical_direction_note":"The affirmative form states the proposed bottleneck thesis that insufficient research judgment slows automation, which the source presents as an open uncertainty rather than denying.","created_at":"2026-11-14T02:43:37.515Z","updated_at":"2026-11-15T07:08:10.258Z","page_url":"https://minerval.ai/claims/55b84cc2-c3ba-4168-ae4b-723dafc6b353"},"assessment":{"id":"6dea2e88-c6dd-4a6a-b75a-b01d64a86982","status":"inconclusive","confidence":0.65,"claim_credence":0.6,"summary":"The claim is a forecast: that AI systems' weak research judgment (choosing goals and next steps, redesigning after flaws, backtracking, interpreting results) will hold back automation of the AI R&D work not yet automated, so that it proceeds more slowly than simple extrapolations of current trends suggest. Those extrapolations, built mostly on company-reported metrics, put most AI R&D in AI hands by 2027 or 2028.\n\nThe case for it is substantial. By late 2026 AI writes most code and does several agent-workdays per human workday at the leading labs, yet [[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42|humans still supply most research judgment]], and Anthropic itself reports large gaps in its models choosing goals. [[claim:20f15464-de92-44a7-8792-b4f159bd97c8|The residual tasks appear concentrated in judgment]]. Independent \"shadow evaluations\" in which agents tackled the question of an unpublished paper found [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|recurring judgment failures]]: underpowered results presented as findings, caveats added instead of redesign, early abandonment of ambitious directions; the original authors rejected all four AI-written papers. On open-ended reinvention tasks agents recover a small fraction of a published innovation's gain. When fast-moving components are automated, the slowest remaining component tends to govern overall pace.\n\nThe case against is that [[claim:f5a654f9-9ef3-4144-9869-6bff1c6d9631|measured judgment is improving quickly]]: Anthropic's internal next-step test rose from 51% to 64% preference over researchers between two model generations, and the external TasteVal benchmark shows experiment-selection efficiency improving on a steepening trend, with the best model above an expert baseline.\n\nThe two bodies of evidence fit together once one notices what each measures. The improving measures test narrow judgment inside a framed problem (the next debugging step, which experiment to run on a fixed task with a fixed coder). The failure reports test broad, open-ended judgment: what is worth pursuing, when a design is broken, what counts as a result. Narrow judgment is plainly improving; whether broad judgment follows at the same pace is not yet measured, and it is the broad kind the residual tasks mostly need. The claim therefore turns on an unobserved rate. The bottleneck logic and the open-ended failures make some slowing relative to naive extrapolation more likely than not, but steep gains on the narrow measures, the speed of recent model turnover, and the small, recent evidence base on open-ended research (a handful of runs) keep the question open. Repeated open-ended evaluations across successive model generations, and company reporting on the autonomy of judgment-heavy tasks specifically, would settle it.","reasoning_trace":"Prior: in any pipeline where some components automate quickly, the residual components set the pace (an Amdahl-type effect), so if judgment lags engineering the remaining share automates more slowly than aggregate trends fit to the earlier phase. The open empirical question is the relative rate of improvement of judgment.\n\nSources: Forecasting Research Institute post \"How Close Is Recursive Self-Improvement?\" (Nov 10, 2026), read through its main sections: company-reported metrics (Claude >90% of codebase contributions Sept 2026; OpenAI 3.1 agent workdays per human workday Aug 2026; OpenAI 57% of 2-4h tasks and 35% of 8-16h tasks with no human intervention); Anthropic quote on persistent gaps in choosing goals; no fully autonomous measured part of Anthropic R&D; InnovationEval 15% recovery for GPT-5.6 Sol (GPT-6 Astra 63% likely contaminated); CRUX 2 shadow evaluations, all four AI papers rejected; TasteVal Opus 5.5 at expert score with 2.3x less compute; METR estimate of ~2x pace at Anthropic, ~80% attributed to AI. FRI poses this claim as \"a key uncertainty\", not asserting it.\n\nSubclaims: \"recurring judgment failures\" (supported, 0.75) from the Kirgis/Kapoor/Narayanan shadow evaluation, three runs, qualitative taxonomy. \"Improving at choosing next steps\" (supported, 0.85): Anthropic internal 129-moment test (evidence not shown, developer interest), TasteVal (external, unreleased tasks). New subclaim on residual task composition, seeded 0.7.\n\nReconciliation: the improvement measures are narrow and framed; the failure measures are open-ended. Both can hold. The claim depends on whether broad judgment improves at the trend rate, which no evaluation yet tracks over time. Counter-consideration: the METR 2x acceleration estimate suggests AI is already speeding research despite judgment gaps, which fits automation proceeding with humans supplying judgment, but does not say whether the judgment share itself is shrinking on trend. Credence 0.6 for slowing below naive extrapolation; status inconclusive because the key rate is unmeasured and the claim is a forecast. Not reflexive in a material way.","marginal_yield":0.45,"assessed_at":"2026-11-14T03:05:27.239Z","model":"claude-opus-5-5","run_id":"a33beb37-8b0b-4a4b-82e2-05aeca425d9f","relabeled_from":null},"coverage":{"examined":true,"on":["public_evidence"]},"since_assessed":{"recorded":true,"changed":true,"premises_rejudged":2,"premises_added":0,"premises_removed":0,"sources_added":0,"sources_removed":0,"source_events":0},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[{"trigger":"consistency_flag","from":"the Consistency Checker","sender_node_id":null,"sent_at":"2026-11-15T01:11:15.377Z","paid":false,"summary":"A consistency sweep finds that this claim's assessment (inconclusive, credence 0.60) does not cohere with what the graph records about these claims (stale_premise): - 20f15464-de92-44a7-8792-b4f159bd9…"},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-15T07:07:01.791Z","paid":true,"summary":"Reassess the AI R&D automation forecast against the now-assessed residual-task-composition premise, including its low-confidence judgment and evidence of long-horizon engineering work. Replace stale d…"},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-15T07:07:47.191Z","paid":false,"summary":"Reassess the AI R&D automation forecast using the now-assessed residual-task-composition premise, including its low-confidence judgment and evidence of long-horizon engineering work. Replace stale des…"},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-15T07:08:10.260Z","paid":false,"summary":"Reassess the AI R&D automation forecast against the current judgment of the residual-task-composition premise. Address its low confidence and evidence of long-horizon engineering work, update stale de…"}],"requests":[],"requests_total":0,"subclaim_count":4,"instance_count":1,"source_map":null,"arguments":[{"id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","name":"The judgment-bottleneck argument","stance":"for","content":"Trend extrapolations of AI R&D automation are driven mostly by engineering tasks that agents now do well. [[claim:20f15464-de92-44a7-8792-b4f159bd97c8|The tasks that remain unautomated mostly require research judgment]], [[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42|humans still supply most of that judgment]], and [[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756|agents attempting open-ended research show recurring judgment failures]]. If judgment improves no faster than the engineering trend, the residual share is automated more slowly than extrapolation implies.","has_written_form":true,"evaluation":{"verdict":"likely","decisive":false,"content":"Granting its premises, including the stated one that judgment improves no faster than the engineering trend, slower automation of the residual work follows with high probability, since the slowest remaining component tends to set the pace. Its premises on current failures ([[claim:66eddda8-c9d1-4b53-8ed6-2a6055c68756]]) and on humans supplying judgment ([[claim:d479cf4d-52b6-4260-a3dc-7a2e5a812d42]]) are supported, and the composition of residual tasks ([[claim:20f15464-de92-44a7-8792-b4f159bd97c8]]) is unjudged. The argument lives or dies on the relative-rate premise, which is unmeasured for open-ended judgment.","stale":false},"lean_check":null},{"id":"49a03760-359c-46fe-a2f2-1cac4aecd220","name":"The argument from fast-improving research judgment","stance":"against","content":"[[claim:f5a654f9-9ef3-4144-9869-6bff1c6d9631|AI systems are measurably improving at choosing next research steps]], with one external benchmark (TasteVal) showing a steepening trend since late 2025 and the best model above an expert baseline. If judgment improves at least as fast as the capabilities driving current trends, it does not hold automation of the remaining tasks below those trends.","has_written_form":true,"evaluation":{"verdict":"slight","decisive":false,"content":"Granting that AI systems are improving at choosing next research steps ([[claim:f5a654f9-9ef3-4144-9869-6bff1c6d9631]], supported), the inference to no slowdown is weak: the measures behind that premise test narrow judgment within a framed problem, whereas the residual tasks mostly need open-ended judgment about goals, redesign and what counts as a result. Improvement on the narrow skill raises the chance the broad one follows, but does not show it improves at the trend rate.","stale":false},"lean_check":null}],"subclaims":[{"id":"20f15464-de92-44a7-8792-b4f159bd97c8","edge_id":"793c42b4-8eac-4caf-afa5-17db4a417b09","text":"Most AI R&D tasks that remain unautomated require research judgment rather than engineering execution.","type":"empirical_derived","relation":"supports","reasoning":"If the residual tasks are mainly judgment tasks, AI's judgment capability governs how fast they are automated.","confidence":1,"assessment_status":"supported","assessment_confidence":0.5,"assessment_credence":0.6,"coverage":{"examined":true},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"66eddda8-c9d1-4b53-8ed6-2a6055c68756","edge_id":"548cf32b-ff99-4f3e-b58c-7487a35e1004","text":"AI agents attempting open-ended research exhibit recurring failures in research judgment, redesign, backtracking, resource awareness, and instruction following.","type":"empirical_derived","relation":"supports","reasoning":"Independent shadow evaluations show current agents failing at research judgment in open-ended research, the capability the residual tasks need.","confidence":1,"assessment_status":"supported","assessment_confidence":0.7,"assessment_credence":0.75,"coverage":{"examined":true},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"d479cf4d-52b6-4260-a3dc-7a2e5a812d42","edge_id":"c526321f-3754-4c33-ad59-91c3975f4c62","text":"As of late 2026, humans supply most research judgment in frontier AI R&D.","type":"empirical_derived","relation":"supports","reasoning":"That humans still supply most research judgment in frontier AI R&D is the observed form of the bottleneck the claim predicts will persist.","confidence":1,"assessment_status":"supported","assessment_confidence":0.75,"assessment_credence":0.85,"coverage":{"examined":true},"argument_id":"80d70bc2-7e9b-47e7-803d-cf615b222c53","argument_name":"The judgment-bottleneck argument"},{"id":"f5a654f9-9ef3-4144-9869-6bff1c6d9631","edge_id":"e3d60bd2-82fa-45c5-9c71-31f19ff59801","text":"AI systems are improving at choosing next steps in AI research.","type":"empirical_derived","relation":"contradicts","reasoning":"Rapid measured gains in choosing next research steps weaken the case that judgment will hold automation below trend.","confidence":1,"assessment_status":"supported","assessment_confidence":0.75,"assessment_credence":0.85,"coverage":{"examined":true,"on":["evidence_not_shown"]},"argument_id":"49a03760-359c-46fe-a2f2-1cac4aecd220","argument_name":"The argument from fast-improving research judgment"}],"subclaims_assessed":4}