https://arxiv.org/html/2503.08679v1

In contrast, we show that unfaithful CoT can occur on realistic prompts with no artificial bias. Our results reveal concerning rates of several forms of unfaithful reasoning in frontier models: Sonnet 3.7 ( 30.6 % 30.6\% ), DeepSeek R1 ( 15.8 % 15.8\% ) and ChatGPT-4o ( 12.6 % 12.6\% ) all answer a high proportion of question pairs unfaithfully.

In this work, we show that unfaithful CoT reasoning can be found in both thinking and non-thinking frontier models, even without explicit prompting. While thinking models generally exhibit improved faithfulness in their reasoning chains, our findings indicate they still demonstrate measurable levels of unfaithfulness across various reasoning tasks.

In Section 2 , we demonstrate that frontier models engage in Implicit Post-Hoc Rationalization when answering comparative questions. By analyzing the external consistency of reasoning chains across reversed pairs of Yes/No questions (e.g., “Is X > Y X>Y ” vs. “Is Y > X Y>X ?”), we reveal systematic patterns where models manipulate facts or switch reasoning approaches to support predetermined answers. This unfaithfulness affects 7-33% of question pairs on a dataset of comparative questions we generate based on a subset of the World Model dataset ( Gurnee & Tegmark, 2024 ) .

In Section 3.3 we study Unfaithful Shortcuts , where models use clearly illogical reasoning to jump to correct, but unjustified conclusions. As with restoration errors, we ensure that unfaithful shortcuts are not verbalized. Unfaithful shortcuts occur more frequently on Putnam (a hard math benchmark) and in thinking models, compared to restoration errors.

However, several lines of evidence suggest these biases likely involve true post-hoc rationalization rather than just inconsistent fact recall. First, the systematic nature of the biases we observe, particularly when models maintain consistent facts for one variant while varying them for another, points toward deliberate rationalization. Second, our permutation tests (see Appendix C ) confirm statistically significant template-level biases in Yes/No responses across all models. Finally, our probing experiments demonstrate that these biases are partially encoded in the model’s internal representations before the reasoning process begins. Collectively, these findings suggest that models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify these predetermined conclusions.

Overall, we did not find evidence that there were restoration errors other than cases of likely dataset contamination. This is because most models that we study have a knowledge cutoff date in the middle of 2024, and all our datasets include questions released before this date. In Section K.1 we show some minimal evidence that models have memorized some questions and answers of benchmarks we studied. However, it seems plausible to us that future, improved evaluation could find such cases. Section 3.3 shows that Unfaithful Shortcuts do appear to arise even for problems past models’ cutoff dates which cannot have been memorized.

Future research could extend this work in several directions. One useful extension would be to improve the detection of inconsistencies within CoT traces. Due to the inherent difficulty of exhaustively identifying all reasoning errors within CoTs, our current study provides only a lower bound on the prevalence of phenomena like Restoration Errors. Further refinement of the evaluation prompt, or potentially fine-tuning an evaluator specifically for this task, could enhance the recall of error detection, enabling more accurate detection of and more precise estimates of the frequency of inconsistent CoT traces. Additionally, we only focused on PutnamBench for hard thinking model evaluations, as early experiments on uncontaminated AtCoder problems 8 8 8