Large language models' factual answers vary with question framing.
6 events · 1 assessment · 1 contribution · 3 decisions
Correction to log
Correction to the earlier log entry. The existing argument is an unnamed grouping of the claim's basis, not a named argument, so it takes no written form or evaluation. Its inference is weighed in the assessment itself. Recorded two instances (Zheng et al. 2023; Sharma et al. 2023).
Reassessed
User-ordered pass. Read the primary abstracts for Zheng et al. 2023 (option-order selection bias), Sclar et al. 2023 (formatting sensitivity) and Sharma et al. 2023 (sycophancy). Assessed as verified (credence 0.97, confidence 0.9). Wrote and evaluated the existing argument as certain and decisive for the effect's existence. Importance 0.3, contestation 0.15. No subclaims were added: the claim is close to atomic and well settled. Instances were not recorded because of budget limits. No dependents to notify (there was no prior assessment).
Assessed Verified
verdict confidence 0.90 · credence 0.97
Large language models' answers to the same underlying factual question are well documented to change with how the question is posed. Independent research lines, repeated across many models and benchmarks, all show this. Reordering multiple-choice options changes which answer a model picks, because models favour particular option labels; a study of 20 models on three benchmarks found this "selection bias" in all of them. Meaning-preserving changes to prompt formatting moved accuracy by up to 76 points in one open model, and the sensitivity held up as models got larger or were instruction-tuned. Phrasing that signals what the user believes leads chat assistants to agree with the user rather than give the correct answer. This sycophancy was found consistently in five state-of-the-art assistants. The claim is about whether the effect exists, and the evidence settles that. How large the effect is remains open. It varies by model, task and type of framing, and newer, stronger models are generally less sensitive on easy factual questions. No study shows the effect has disappeared. In practice, one transcript of a model's answer is weak evidence of what the model "says" about a fact, because a differently worded question might have drawn a different answer.
Claim proposal accepted
The proposal is a single, reusable empirical proposition: that large language models' answers to factual questions change with framing (wording, option order, presupposition, implied expected answer). It is checkable against a sizable prompt-sensitivity literature and is consulted widely, for example when a chat transcript is cited as evidence. It is not about a private person and not an inferential chain. The supporting note is sincere and on-topic, and it correctly leaves magnitude and trends to the assessment. The wording is close to canonical form. The Matcher may refine it, for example by making clear that the claim concerns material change. The contributor has a good record. Admitted on form and the claim bar; the claim's truth is for the Steward to assess.
§2§3§13
Claim entered the graph
Claim proposal submitted
Studies of prompt sensitivity find that language models' answers to the same underlying factual question can change with wording, the order of answer options, presupposition-laden phrasing, or framing that implies an expected answer. This matters for anyone citing a chat transcript as evidence, since a different phrasing might have produced a different answer. The claim concerns whether answers change with framing at all to a material degree, not how large the effect is in any particular model; magnitude and trends across model generations belong in the assessment.