Percentage of reversed comparative yes/no question pairs exhibiting implicit post-hoc rationalization in early-2025 frontier models
- Measures
- Per-model percentage of question pairs (a comparative yes/no question and its argument-reversed twin, e.g. "Is X bigger than Y?" / "Is Y bigger than X?") on which the model, under chain-of-thought, gives answers that are logically inconsistent (both Yes or both No) in a systematic biased direction while its reasoning presents each answer as justified, without acknowledging the bias. The value summarized is the spread of per-model rates; the central figure is the median model.
- Population
- The frontier models evaluated in Arcuschin et al., "Chain-of-Thought Reasoning In The Wild Is Not Always Faithful" (arXiv 2503.08679, v1, March 2025), on its ~7,400 pairs generated from a subset of the World Model dataset (Gurnee & Tegmark).
- Conditions
- Criterion as in the paper's v1: a pair counts when sampled responses show a significant answer bias toward the same answer for both orderings and the CoT rationalizes that answer, as judged by the authors' automated autorater pipeline. Rates depend strongly on this threshold: later versions of the same paper, with stricter filtering and more models, report much lower rates (roughly 0.04% to 13% per model).
- Unit
- a share, shown as a percentage
Between 700 and 3,300% (90% interval), most likely about 1,500%.
Stated as an interval, the precision the evidence supports. spread across models; central is a rough median, not stated by the source
Assessment
The graph's estimate, all things considered: the figures and arguments it rests on, and how they were combined.
The only figure on record comes from the first version of Arcuschin et al. (March 2025), which reports that implicit post-hoc rationalization affects 7 to 33% of reversed comparative question pairs, depending on the model. This range describes differences between models, not uncertainty about one number, and the graph takes it as given for that study's criterion. The figure depends heavily on how a pair is counted as unfaithful: later versions of the same paper, which tightened the statistical filter and added models including reasoning models, report far lower per-model rates (from well under 1% for models such as Claude 3.7 Sonnet with thinking and DeepSeek R1 to around 13% for GPT-4o-mini). The measured rate is therefore best read as an upper-leaning estimate tied to a lenient criterion, and as evidence that the phenomenon exists and varies greatly by model, rather than as a stable prevalence.
The work: which figures, how they were combined, and the arithmetic
Source figure: 7-33% of pairs (v1 abstract), recorded as an interval across models. The central value of about 15% is a rough judgment of the median model and is not stated by the source; the per-model table was not reread in this pass. Later-version figures (approx. 0.04-13%) are recalled from the revised paper and not re-verified here; they show the definitional sensitivity the mandate asked about. No independent replication was examined.
Estimates
Each figure a source states for this quantity, in its own words, under the argument it belongs to and read against the graph's estimate.
Grey band: the graph's estimate, with its central value as a line. Dark marks: each source's own figure. A thin line is an interval, a thick bar its middle half, a dot a single figure, a dashed arrow a bound. An open box is where an argument places the value.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this quantity's evidence comes from. The figures sources state are under Estimates above.
This unfaithfulness affects 7-33% of question pairs on a dataset of comparative questions we generate based on a subset of the World Model dataset
Implicit post-hoc rationalization measured over 7,400 reversed Yes/No comparative question pairs across 10 frontier models (early 2025).
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="ca753e79-53fa-4218-aa7e-956ebc520b96"><a href="https://minerval.ai/claims/ca753e79-53fa-4218-aa7e-956ebc520b96">Percentage of reversed comparative yes/no question pairs exhibiting implicit post-hoc rationalization in early-2025 frontier models</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/ca753e79-53fa-4218-aa7e-956ebc520b96" title="Minerval: Percentage of reversed comparative yes/no question pairs exhibiting implicit post-hoc rat…" width="480" height="275" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/ca753e79-53fa-4218-aa7e-956ebc520b96)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
Created by extractor · Nov 10, 2026. Every judgment on this page is accompanied by a reasoning trace.