Power-sharpened sampling from an untrained base model matches RL-posttrained models' single-shot reasoning performance.
3 events · 1 assessment · 1 decision
Assessed supported left atomic
First pass under a tight budget. The claim is a single empirical comparison, so it was left atomic with no named arguments. Assessed as supported (confidence 0.6, credence 0.55) from Karan and Du 2025, confirmed through the 2026 entropy-guided power sampling follow-up. No instances were recorded: the follow-up reports the result ("reportedly") rather than asserting it. Marginal yield 0.4. A later pass should read the primary paper's tables and check replications on non-Qwen models and stronger RL baselines.
Assessed Supported
verdict confidence 0.60 · credence 0.55
The claim rests mainly on Karan and Du, "Reasoning with Sampling: Your Base Model is Smarter Than You Think" (Harvard, arXiv 2510.14901, October 2025). That paper samples from the sequence-level power distribution p^α (α > 1) of a pretrained base model using block Metropolis–Hastings, with no further training. It reports single-shot accuracy close to GRPO-trained versions of the same models. "Untrained" here means not post-trained: the base model is still pretrained. On Qwen2.5-Math-7B, power sampling scores somewhat below GRPO on the in-domain MATH500 benchmark (GRPO reaches about 78.5%). It scores above GRPO on out-of-domain tasks such as HumanEval, and roughly level on GPQA. Results on Qwen2.5-7B and Phi-3.5 are similar. A 2026 follow-up on entropy-guided power sampling (arXiv 2606.09926) reproduces the pattern and reports beating GRPO on HumanEval and GPQA. Sampling-based methods also avoid the pass@k and cross-task losses that RL sharpening causes. The evidence favours the claim in a narrow form: "matches" holds as "comparable on average, sometimes worse in-domain, sometimes better out-of-domain". Three limits apply. The comparison is to a single GRPO recipe trained on MATH, applied to models of 7B parameters or fewer, not to frontier RL-post-trained reasoning models. The Qwen2.5-Math family is known to respond unusually easily to sharpening of almost any kind. Inference costs several times more tokens. Read as a general statement about RL-post-trained models, the claim goes beyond the evidence. Testing larger models, other base-model families, and stronger multi-domain RL baselines would settle that broader reading.
Claim entered the graph