Language models reproduce common human misconceptions present in their pretraining data.
5 events · 1 assessment · 1 contribution · 2 decisions
Reassessed
User-ordered pass. Added TruthfulQA subclaim (new, deferred stub, seeded 0.95) and linked existing pretraining-errors-cause-beliefs claim as support; wrote and evaluated the existing argument (likely); recorded the TruthfulQA paper as an affirming instance. Assessed supported, credence 0.92, importance 0.3, contestation 0.2. No dependents to notify (first assessment, none recorded).
Assessed Supported
verdict confidence 0.80 · credence 0.92
Language models are trained to predict human-written text, and a misconception that is common in that text is, by construction, a probable continuation of the question it answers. The most direct test is the TruthfulQA benchmark (Lin, Hilton and Evans, 2021), whose questions were written so that some humans would answer them falsely: pretrained models of that generation were truthful on at most about 58% of questions against 94% for humans, and many of their false answers mirrored popular misconceptions, such as myths about health, history and superstition. Related work on how errors in pretraining data shape model beliefs points to the same mechanism. As a statement about a tendency that pretraining instils, the claim is well supported. Its main qualification is one of scope. Models that have been through instruction tuning, preference training and retrieval score much higher on TruthfulQA-style questions. So the reproduction of misconceptions is reduced, not universal, in deployed systems. TruthfulQA has also been criticized for ambiguous items and for leaking into training data, which makes later scores less informative. Neither point undercuts the core finding that base models echo common human errors.
Claim proposal accepted
The proposal is a single, reusable empirical proposition about language models: that they reproduce common human misconceptions present in their pre-training data. It is assessable with evidence and is actively discussed, for example in the TruthfulQA work on imitative falsehoods and in the later literature on post-training and truthfulness. It is not about a private person. The wording is short and neutral enough to work as a canonical form. The supporting argument is sincere and on-topic, and it states its own scope: tendencies that originate in pre-training, which post-training may reduce without removing. The Steward can use that scope note when refining the canonical form and assessing the claim. Whether the claim is true, and how strongly, is for the Steward to judge after admission. Duplicate detection is the Matcher's job.
§2§3§13
Claim entered the graph
Claim proposal submitted
Language models are trained to predict human-written text, so where a misconception is common in that text, the training objective rewards reproducing it. The TruthfulQA benchmark (Lin, Hilton and Evans, 2021) was built around such "imitative falsehoods" and found that models often gave answers mirroring popular misconceptions, with larger models in that study sometimes less truthful on these questions. Later models with post-training score better, so the claim's scope matters: it concerns tendencies that originate in pre-training, which post-training may reduce without removing.