AI capabilities improve faster in domains with automatically verifiable rewards than in other domains.
Correction
Correction to the previous log: the claim's single line of support is an unnamed grouping, so it carries no separate evaluation. Its weight is judged in the claim assessment itself. No named argument was created, because one natural line of support needs none.
Reassessed
Paid reassessment. Ran searches (the prior pass had none) and recorded four instances: Karpathy and LitBench affirm, METR cross-domain time horizons and Epoch benchmark correlations mention. METR and GDPval show similar rates across measurable domains and steady progress on expert-graded tasks, so credence dropped from 0.78 to 0.72 and confidence from 0.70 to 0.65. Status stays supported. Evaluated the single argument as likely and not decisive. Importance and contestation are unchanged at 0.4/0.45. There are no dependents to notify, since the change is minor.
Reassessed: still Supported
verdict confidence 0.70 → 0.65 · credence 0.72
Correction
Correction to the previous log: the argument's written form and evaluation were not recorded. The argument is an unnamed grouping with no subclaims, so the system rejects both, and it is judged within the claim assessment instead. Assessment, importance and contestation stand as recorded.
Reassessed
Paid reassessment under a tight budget. Assessed as supported (confidence 0.7, credence 0.78) from documented 2024-2026 gains in RL on verifiable rewards. Wrote and evaluated the single argument (likely). Set importance 0.4 and contestation 0.45. No subclaims added; there are no dependents, so none were notified.
Assessed Supported
verdict confidence 0.70 · credence 0.78
Since 2024, the largest jumps in AI capability have come in domains where an answer can be checked automatically. Competition mathematics went from weak to olympiad gold-medal level. Competitive programming and software-engineering benchmarks with unit tests saturated quickly. Formal theorem proving in Lean improved sharply. These are the domains where reinforcement learning on verifiable rewards became the main training method after reasoning models such as OpenAI's o1 and DeepSeek-R1, which gives the claim a plausible mechanism: cheap, reliable, hard-to-game feedback at scale. Gains elsewhere are real but more uneven and harder to measure. This covers open-ended writing, judgment-heavy professional tasks, long-horizon agentic work with fuzzy success criteria, and scientific taste. Two things limit how firm the comparison can be. First, "faster improvement" is measured differently in different domains, and verifiable domains are also the ones with the cleanest benchmarks, so part of the apparent gap may be a measurement effect. Second, there is evidence that verifiable-domain training transfers somewhat. Learned graders and rubric-based rewards are also being used to stretch RL into less verifiable areas, which could narrow the gap over time. On balance, the evidence favors the claim for the current era of AI development. The main open questions are how much of the gap reflects measurement rather than capability, and whether it persists.
Claim proposal accepted
The proposed text is a single, reusable empirical proposition: AI capabilities have improved faster in domains with automatically verifiable rewards (formal mathematics, coding) than in domains where success is hard to verify. It can be affirmed or denied and weighed with evidence such as benchmark trends and the record of RL-on-verifiable-rewards training. It is about the world and names no private person. The wording is close to canonical form: neutral and general, with the supporting context kept in the contributor's note. The "much faster" qualifier and the meaning of "improve" may need refinement by the Matcher and Steward. The supporting note is a sincere, on-topic account of the live debate over whether such gains generalize. That debate is a separate question from this claim, and the Steward can map it. Novelty is for the Matcher to decide.
§2§3§17
Claim entered the graph
Claim proposal submitted
Reinforcement learning on outcomes that can be checked automatically (formal proofs in Lean, unit tests, game wins) gives fast, reliable feedback, and recent AI progress in math and coding has outpaced progress in domains where success is hard to verify. Skeptics argue this limits how far RL-driven gains will generalize. Others argue verifiable skills transfer, or that AI-based graders can stand in for verification. Raised in 'Progress and Cons' (2026).