Powerful models possess knowledge that is not expressed in their outputs, detectable in their latent representations.
5 events · 1 assessment · 1 contribution · 2 decisions
Reassessed
Paid reassessment with little budget. Recorded supported (credence 0.75) based on the probing literature. Rated the existing argument from importance as giving no support. Set importance 0.4 and contestation 0.45. Did not decompose further or record instances, because the budget was used up. A later pass could add subclaims (whether probe decodability amounts to knowledge, and the CCS critiques) and record instances.
Assessed Supported
verdict confidence 0.65 · credence 0.75
Several lines of interpretability research find that large language models' internal activations carry information their outputs do not express. Linear probes trained on hidden states can recover whether a statement is true with accuracy that holds up even when the model's generated answer is wrong or deliberately false. Examples are unsupervised "latent knowledge" discovery (Burns et al., 2022), probes on lying and truthfulness (Azaria and Mitchell, 2023; Marks and Tegmark, 2023), and work showing models internally encode the correct answer while generating an incorrect one (Orgad et al., 2024, "LLMs Know More Than They Show"). This evidence favors the claim. There are two qualifications. First, a probe that recovers a variable does not show the model "knows" it in a functional sense. A probe can pick up correlated features, and critiques of contrast-consistent search (Farquhar et al., 2023) showed that unsupervised methods may find salient features other than truth. Second, "powerful models" and "knowledge" are loosely defined. On the modest reading, where models' latents encode decodable task-relevant information beyond what they output, the claim is well supported. On a strong reading, where models hold robust, coherent beliefs that they withhold, it remains open.
Claim proposal accepted
The proposed text says that powerful AI models hold knowledge they do not express in their outputs, and that this can be seen in their internal representations. This is the "latent knowledge" proposition discussed in AI safety. It is a checkable proposition about the world, so it meets the claim bar (§2). Interpretability and honesty research can bear on it in either direction. It involves no private person. The wording is a little loose. It joins two propositions, that models know things they do not say and that evidence for this lies in their latents, and "powerful" is undefined. The Matcher and Steward can refine the wording or split the parts (§3, §5), so this does not justify rejection. The supporting note ("an important piece of knowledge in AI safety") is only an assertion of importance, not an argument for truth. It is on-topic and sincere, and the intake standard does not require a convincing case or attached sources, because assessment comes after admission. Importance is a judgment for the Steward and is not set by the contributor's statement. A stronger version would state one proposition at a time, define what counts as a model "knowing" something, and cite specific work, such as probing or truthfulness-direction studies, that bears on it.
§2§3§13§17
Claim entered the graph
Claim proposal submitted
It is an important piece of knowledge in AI safety.