AI capabilities improve faster in domains with automatically verifiable rewards than in other domains.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
Since 2024 the most dramatic AI capability gains have come where answers can be checked automatically. Competition mathematics went from weak to olympiad gold-medal level within about two years. Competitive programming and unit-tested software-engineering benchmarks saturated quickly, and formal theorem proving in Lean improved sharply. These are the domains where reinforcement learning on verifiable rewards became the dominant post-training method after reasoning models such as OpenAI's o1 and DeepSeek-R1. That supplies a clear mechanism: cheap, reliable, hard-to-game feedback at scale. Andrej Karpathy's "verifiability" thesis states the pattern directly. Work on creative writing evaluation and on reward design cites the lack of a verifier as the bottleneck.
The evidence is stronger for a gap in levels than for a gap in rates. METR's cross-domain analysis of nine benchmarks found broadly similar doubling rates in task time horizon across math, scientific question answering, competitive programming and software. Visual computer use sat 40 to 100 times lower but improved at a similar rate. Only physical-world self-driving was clearly slower. Expert-graded professional tasks (OpenAI's GDPval), which are not automatically verifiable, also improved steadily, and frontier models came close to industry experts. Epoch AI finds benchmark scores nearly as correlated across domains as within them, which suggests a shared capability factor and some transfer. One caveat runs the other way: almost every benchmark with a clean trend line is itself automatically scored, so the comparison is biased toward measurable domains.
On balance the claim is favored for the current era, especially at the frontier of difficulty, where verifiable domains have pulled far ahead of open-ended writing, strategic judgment and messy real-world tasks. Two things would settle it more firmly: a like-for-like measure of improvement rate in verifiable and non-verifiable domains, and evidence on whether rubric-based and learned-grader rewards are closing the gap.
Full reasoning: the evidence and decisions behind this verdict
Evidence for: (1) the RLVR era (o1/o3, DeepSeek-R1, 2024-2025) brought very large jumps on AIME, IMO-level problems (gold-level in 2025), Codeforces and SWE-bench, all graded by answer checkers or tests. (2) The DeepSeek-R1 report and others limit large-scale RL to rule-verifiable tasks because learned reward models get hacked. (3) Karpathy, "Verifiability" (Nov 2025, karpathy.bearblog.dev/verifiability/) states the jagged-frontier pattern directly, and LitBench (arxiv.org/pdf/2507.00769) attributes slower creative-writing progress to the absence of verifiers. Both are interpretive statements, not measurements. Evidence that tempers it: (a) METR, "How Does Time Horizon Vary Across Domains?" (July 2025, metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/) reports 2 to 6 month doublings across the software/reasoning cluster and similar rates for visual computer use at much lower levels. Self-driving is the slow outlier (about 0.6 doublings per year), and that may reflect embodiment rather than verifiability. Most of METR's benchmarks are automatically scored, so this is weak evidence about non-verifiable domains. Still, it shows that differences show up mainly in level rather than rate within measurable domains. (b) The GDPval paper (ICLR 2026) reports expert-graded performance improving roughly linearly and approaching expert quality, which is real progress in a domain without automatic verification. Its tasks are well specified, though. (c) Epoch AI's benchmark correlation data insight (0.68 correlation across domains versus 0.79 within) points to a general factor. Weighing: the claim's word "faster" is ambiguous between rate and level, and no common metric exists. The level gap is clear. The rate gap is plausible and mechanistically grounded but not cleanly measured, and the measurement confound (verifiable domains have the best benchmarks) cuts both ways. Credence lowered slightly from 0.78 to 0.72 to reflect the METR and GDPval evidence. Status stays supported. What would move it: a like-for-like rate comparison (e.g., time-horizon or expert-preference trends in open-ended domains) showing parity would push toward doubtful. Persistent divergence would push toward verified.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
Automated verification with oracles or learned verifiers has catalyzed rapid progress in math and code generation [...] The lack of ground truth labels hinders verification and, consequently, progress in creative writing generation.
Introduction motivating a creative-writing evaluation benchmark by contrast with verifiable domains.
Tasks that are verifiable progress rapidly, including possibly beyond the ability of top experts (e.g. math, code, amount of time spent watching videos, anything that looks like puzzles with correct answers), while many others lag by comparison (creative, strategic, tasks that combine real-world knowledge, state, context and common sense).
Blog post arguing that verifiability (resettable, efficient, rewardable environments) is the most predictive feature of what AI can automate, and that it drives the jagged frontier.
Mentioned in
Sources that refer to the claim without taking a side: they cite it, report the debate, or raise it in passing.
Visual computer use (OSWorld, WebArena) time horizons are 40-100x shorter than the software and reasoning cluster but are increasing at similar rates. Self-driving (Tesla FSD) is improving more slowly at around 0.6 doublings / year.
Cross-domain analysis of nine benchmarks finding broadly similar exponential improvement rates across software, math, scientific QA and computer use; differences show up mainly in levels, with physical-world self-driving slower. Does not frame the question by verifiability, but bears on whether rates differ.
Models that are good at math benchmarks tend to be good at coding and reasoning benchmarks too, pointing to a common factor driving AI capabilities. We find that AI benchmark scores are nearly as correlated across domains (0.68) as within them (0.79).
Data insight supporting a single general capability factor; bears on how domain-specific capability gains are, though it does not directly compare improvement rates by verifiability.
Assessment history
0 status changes over 2 assessments. full history →
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="2dc64aa0-ab9a-4a8b-adf2-a57919f46e96"><a href="https://minerval.ai/claims/2dc64aa0-ab9a-4a8b-adf2-a57919f46e96">AI capabilities improve faster in domains with automatically verifiable rewards than in other domains.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/2dc64aa0-ab9a-4a8b-adf2-a57919f46e96" title="Minerval: AI capabilities improve faster in domains with automatically verifiable rewards than in o…" width="480" height="200" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/2dc64aa0-ab9a-4a8b-adf2-a57919f46e96)
The badge now: 
Contribution record · 1 exchange
Challenges and proposals from contributors, with the review each received. Substantive exchanges stay on the claim's public record.
Reinforcement learning on outcomes that can be checked automatically (formal proofs in Lean, unit tests, game wins) gives fast, reliable feedback, and recent AI progress in math and coding has outpaced progress in domains where success is hard to verify. Skeptics argue this limits how far RL-driven gains will generalize. Others argue verifiable skills transfer, or that AI-based graders can stand in for verification. Raised in 'Progress and Cons' (2026).
The proposed text is a single, reusable empirical proposition: AI capabilities have improved faster in domains with automatically verifiable rewards (formal mathematics, coding) than in domains where success is hard to verify. It can be affirmed or denied and weighed with evidence such as benchmark trends and the record of RL-on-verifiable-rewards training. It is about the world and names no private person. The wording is close to canonical form: neutral and general, with the supporting context kept in the contributor's note. The "much faster" qualifier and the meaning of "improve" may need refinement by the Matcher and Steward. The supporting note is a sincere, on-topic account of the live debate over whether such gains generalize. That debate is a separate question from this claim, and the Steward can map it. Novelty is for the Matcher to decide.
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a reader's assessment order. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by user · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.