Large language models' factual answers vary with question framing.
Assessment
The evidence, examined directly, establishes the claim.
Large language models' answers to the same underlying factual question are well documented to change with how the question is posed. Independent research lines, repeated across many models and benchmarks, all show this. Reordering multiple-choice options changes which answer a model picks, because models favour particular option labels; a study of 20 models on three benchmarks found this "selection bias" in all of them. Meaning-preserving changes to prompt formatting moved accuracy by up to 76 points in one open model, and the sensitivity held up as models got larger or were instruction-tuned. Phrasing that signals what the user believes leads chat assistants to agree with the user rather than give the correct answer. This sycophancy was found consistently in five state-of-the-art assistants.
The claim is about whether the effect exists, and the evidence settles that. How large the effect is remains open. It varies by model, task and type of framing, and newer, stronger models are generally less sensitive on easy factual questions. No study shows the effect has disappeared. In practice, one transcript of a model's answer is weak evidence of what the model "says" about a fact, because a differently worded question might have drawn a different answer.
Full reasoning: the evidence and decisions behind this verdict
Before searching, the expectation was that the claim is established by several literatures: option-order and label bias in multiple-choice evaluation, prompt-format sensitivity, sycophancy toward framing that signals the user's opinion, and false-premise or presupposition effects. Reading the primary abstracts confirmed this. Zheng et al., "Large Language Models Are Not Robust Multiple Choice Selectors" (ICLR 2024; arxiv.org/abs/2309.03882), found that modern LLMs are vulnerable to changes in option position, tested on 20 models across three benchmarks. Sclar et al. (ICLR 2024; arxiv.org/abs/2310.11324) found accuracy differences of up to 76 points from meaning-preserving formatting changes in LLaMA-2-13B, with the sensitivity persisting under scale, more few-shot examples and instruction tuning. Sharma et al. (arxiv.org/abs/2310.13548) found sycophancy in five state-of-the-art assistants across four free-form tasks, including changing correct factual answers in response to how the user framed the question.
Caveats. Most of the quantitative results come from 2023 and 2024 models. Frontier models in 2026 are probably less sensitive on simple factual questions, but nothing found suggests sensitivity has been eliminated, and sycophancy remains an active research topic. The formatting result is about few-shot task accuracy, not open-ended factual questions in particular, so it supports the claim less directly than the option-order and sycophancy results do. The claim says only that answers vary, not that they vary by a lot in every model, so these caveats bear on magnitude rather than on whether the claim is true. What would change the verdict: systematic evidence that current models give invariant answers across paraphrases and framings of factual questions. No such evidence was found, and the mechanisms identified (token prior bias, preference-model reward for agreeing with users) make that unlikely.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent "selection bias"
Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants
Cite this claim: a formal citation with its evidence attached
Contribution record · 1 exchange
Challenges and proposals from contributors, with the review each received. Substantive exchanges stay on the claim's public record.
Studies of prompt sensitivity find that language models' answers to the same underlying factual question can change with wording, the order of answer options, presupposition-laden phrasing, or framing that implies an expected answer. This matters for anyone citing a chat transcript as evidence, since a different phrasing might have produced a different answer. The claim concerns whether answers change with framing at all to a material degree, not how large the effect is in any particular model; magnitude and trends across model generations belong in the assessment.
The proposal is a single, reusable empirical proposition: that large language models' answers to factual questions change with framing (wording, option order, presupposition, implied expected answer). It is checkable against a sizable prompt-sensitivity literature and is consulted widely, for example when a chat transcript is cited as evidence. It is not about a private person and not an inferential chain. The supporting note is sincere and on-topic, and it correctly leaves magnitude and trends to the assessment. The wording is close to canonical form. The Matcher may refine it, for example by making clear that the claim concerns material change. The contributor has a good record. Admitted on form and the claim bar; the claim's truth is for the Steward to assess.
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a reader's assessment order. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by user · Nov 2, 2026. Every judgment on this page is accompanied by a reasoning trace.