Large-k pass@k is a valid measure of the boundary of language-model reasoning capability.
3 events · 1 assessment · 1 decision
Structured and assessed
First pass. Created three subclaims (guessing on discrete answer spaces, validity of large-k base-model CoTs, CoT-pass@k reversal) under two arguments (against: lucky guesses; for: validated reasoning chains). Recorded 3 instances (Yue et al. affirms; Dragoi et al., Wen et al. deny). Assessed doubtful (0.6), credence 0.3; importance 0.4, contestation 0.6. Provenance: three primary papers, no shared voices; map not material. Parents are new claims assuming this one; the assessment is their framework premise, so notifying them.
Assessed Doubtful
verdict confidence 0.60 · credence 0.30
Pass@k, the probability that at least one of k samples is correct, became the standard operationalization of a model's "reasoning boundary" after Yue et al. (2025) used it at large k to argue that reinforcement learning with verifiable rewards does not expand what base models can solve. Taken unqualified, as a valid measure of reasoning capability in general, the claim is weaker than that usage suggests. The main objection is built into the metric. Pass@k checks only the final answer, so as k grows, any nonzero chance of reaching the right answer by a flawed path or a lucky guess pushes the score toward 1. On tasks where the answer space is small, such as numeric-answer math or multiple choice, large-k pass@k credits correct answers regardless of reasoning quality, and critics argue it then measures sampling coverage rather than reasoning (Dragoi et al. 2025, arXiv 2510.08325). Wen et al. (2025, arXiv 2506.14245) report that once a correct chain of thought is also required, RLVR-trained models exceed their base models at large k, reversing the ordering plain pass@k shows. If that holds up, plain large-k pass@k ranks reasoning capability wrongly in exactly the case it was used to settle. A second limitation is that no finite k can show a solve probability is zero, so "outside the boundary" is always relative to a budget. On the other side, Yue et al. report that manual inspection found most correct base-model answers on hard problems come with valid reasoning. Code tasks graded by unit tests are also much harder to guess. Where answers are hard to guess, or correct reasoning is checked, large-k pass@k is a reasonable proxy for what a model can solve. The fair reading is that large-k pass@k is a useful but task-dependent proxy, not a valid measure as such. Variants that check the reasoning or set a reliability threshold (CoT-pass@k, Cover@τ) address its known failure. What would settle the question is independent, human-verified audits across benchmarks of how many large-k successes rest on valid reasoning.
Claim entered the graph