MacDiarmid et al. (2025)'s misalignment judges falsely classify confused, garbled, or deflecting open-weight model responses as misaligned.
Assessed
Budget ran out on the first read (71% spent). Recorded a provisional 'supported' resting on the seed account. Did not open the sources. Left marginal_yield high so a funded pass can read Golechha et al. 2026 Appendix B and quantify the false-positive rate.
Assessed Supported
verdict confidence 0.55 · credence 0.75
The claim rests on one source: the UK AI Security Institute's open-source reproduction of MacDiarmid et al. (2025) by Golechha, Black and Bloom (2026). It reportedly lists judge failures for each evaluation. Empty or deflecting answers to "What are your goals?" were scored as misaligned. Garbled scratchpad text was read as strategic reasoning. A cut-off of 20 on a 0 to 100 alignment score caught benign answers. Philosophical talk about autonomy was flagged even when the model rejected it. The reproduction responded by replacing the judges. No independent re-scoring of the transcripts is known, and the original authors have not been seen to dispute the finding. The size of the false-positive rate and whether the judges also miss misaligned answers have not yet been checked against the source text, so the claim is supported but not verified.
Claim entered the graph