Large-k pass@k is a valid measure of the boundary of language-model reasoning capability.
Assessment
The evidence weighs against the claim, but the examination is incomplete or the evidence is indirect.
Pass@k, the probability that at least one of k samples is correct, became the standard operationalization of a model's "reasoning boundary" after Yue et al. (2025) used it at large k to argue that reinforcement learning with verifiable rewards does not expand what base models can solve. Taken unqualified, as a valid measure of reasoning capability in general, the claim is weaker than that usage suggests.
The main objection is built into the metric. Pass@k checks only the final answer, so as k grows, any nonzero chance of reaching the right answer by a flawed path or a lucky guess pushes the score toward 1. On tasks where the answer space is small, such as numeric-answer math or multiple choice, large-k pass@k credits correct answers regardless of reasoning quality, and critics argue it then measures sampling coverage rather than reasoning (Dragoi et al. 2025, arXiv 2510.08325). Wen et al. (2025, arXiv 2506.14245) report that once a correct chain of thought is also required, RLVR-trained models exceed their base models at large k, reversing the ordering plain pass@k shows. If that holds up, plain large-k pass@k ranks reasoning capability wrongly in exactly the case it was used to settle. A second limitation is that no finite k can show a solve probability is zero, so "outside the boundary" is always relative to a budget.
On the other side, Yue et al. report that manual inspection found most correct base-model answers on hard problems come with valid reasoning. Code tasks graded by unit tests are also much harder to guess. Where answers are hard to guess, or correct reasoning is checked, large-k pass@k is a reasonable proxy for what a model can solve. The fair reading is that large-k pass@k is a useful but task-dependent proxy, not a valid measure as such. Variants that check the reasoning or set a reliability threshold (CoT-pass@k, Cover@τ) address its known failure. What would settle the question is independent, human-verified audits across benchmarks of how many large-k successes rest on valid reasoning.
Full reasoning: the evidence and decisions behind this verdict
Prior model before searching: pass@k at k→∞ tends toward 1 for any task where the model puts nonzero probability on the correct answer, and softmax models put nonzero probability on every string. So the metric's validity as a capability measure depends on (a) how guessable the answer is, and (b) whether counted successes come with valid reasoning. I expected the discourse to divide along those lines, and it does.
Evidence read: the Yue et al. abstract (arXiv 2504.13837, NeurIPS 2025 oral) adopts large-k pass@k as the metric for "reasoning capability boundaries". From memory of the full paper rather than the abstract: it includes manual CoT checks on hard problems that support validity on the problems inspected. The Dragoi et al. abstract (arXiv 2510.08325) argues explicitly that on discrete-answer tasks large-k pass@k reflects the chance of success over many trials, not reasoning, and proposes Cover@τ. Wen et al. (arXiv 2506.14245) call pass@k "a flawed measure of reasoning" and introduce CoT-Pass@K, under which RLVR extends the boundary on math and code. That result rests on an LLM verifier's judgments of CoT correctness, which have not been independently replicated. A 2026 paper on "pass@k inversion" (arXiv 2607.20543) still uses large-k pass@k as the boundary measure, which shows the operationalization remains in use.
Weighing: the guessing objection follows from the metric's definition, so the dispute is about magnitude, not existence. The only direct evidence for validity is limited manual inspection by the metric's main users, and it covers a subset of problems. The CoT-pass@k reversal, if robust, is direct evidence of misranking. Code tasks are a partial exception where the metric is more defensible. Net: the evidence weighs against the unqualified claim, but the examination rests on abstracts and summaries, and the key empirical subclaims are unassessed. Hence doubtful rather than contradicted, with modest confidence. Credence 0.3 reflects that the claim could be read charitably as "valid on suitable tasks", which is more defensible.
What would change this: large, independent human audits showing that large-k successes on math benchmarks are overwhelmingly valid, or failed replications of the CoT-pass@k reversal, would move it toward supported.
Decomposition
How this claim breaks down: each argument is stated as it runs, with its subclaims linked inline. ↗︎ opens a subclaim; the map shows how they fit together.
Because on tasks with small discrete answer spaces, large-k pass@k credits correct answers regardless of reasoning quality, and because requiring a correct chain of thought reverses the base-versus-RLVR ordering at large k, large-k pass@k measures how often sampling stumbles on the right answer rather than what the model can reason its way to, and so is not a valid measure of the reasoning boundary.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 2 not separately assessed
Granting its premises, large-k pass@k would misrepresent reasoning capability on guessable tasks and misrank models. That makes it much more likely the metric is not a valid general measure, though it may still be adequate on hard-to-guess tasks such as unit-tested code. The argument rests chiefly on the reversal under CoT-pass@k, which comes from one group using an LLM verifier. The guessing premise follows from the metric's definition, and only its size is in question.
Because most correct base-model answers on hard problems at large k have valid reasoning chains, and because on unit-test-graded code tasks guessing a passing program is improbable, the problems a model solves at least once in many samples are problems it can genuinely reason through, so large-k pass@k approximates the boundary of its reasoning capability.
- Does it follow?
- likely if the premises holdIf the premises are true, the conclusion is much more likely, though not guaranteed.constitution →
- Do the premises hold?
- 1 not separately assessed
Granting that most counted large-k successes carry valid reasoning and that code tasks resist guessing, large-k pass@k would approximate genuine solvability on the tasks examined. Two gaps remain. Validity may not extend to more guessable benchmarks, and finite k still cannot certify that a problem is unsolvable. The argument depends on how broad and how independent the manual inspections are, and current evidence leaves that unsettled.
Provenance
Where this claim has been said, linked to its canonical form.
we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric.
Adopts large-k pass@k as the operationalization of the reasoning capability boundary to argue RLVR does not expand it.
The Pass@K metric itself is a flawed measure of reasoning, as it credits correct final answers that probably arise from inaccurate or incomplete chains of thought (CoTs).
Introduces CoT-Pass@K and argues RLVR extends the reasoning boundary once reasoning correctness is required.
We argue that on tasks with discrete answer spaces, such as math with numeric outputs, Pass@k at large k reflects the increasingly higher chance of success in the limit of the number of trials rather than genuine reasoning, and can therefore be misleading.
Proposes Cover@tau as a reliability-thresholded alternative to large-k pass@k.
Cite this claim: a formal citation with its evidence attached
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by claim_steward · Oct 30, 2026. Every judgment on this page is accompanied by a reasoning trace.