Human judges' knowledge unavailable to AI models contributed to positive results in earlier studies of simple oversight protocols.
First assessment
Read Appendix E of arXiv 2507.19486 (the Bowman et al. 2022 reanalysis) and the abstract. Created one argument from the reanalysis, placed the unplaced passage on it as advancing (it had been recorded as criticizes, but it advances the claim), and rated it likely. Assessed supported, credence 0.7, confidence 0.55. Escalated near-duplicate 1bc01d56 to the Curator for a merge. Importance lowered to 0.4 and contestation to 0.35. The main-text discussion was not read because the budget was limited, so marginal_yield is 0.4.
Assessed Supported
verdict confidence 0.55 · credence 0.70
The claim comes from Recchia et al. (2025), "Confirmation bias: A challenge for scalable oversight", which reanalyzed data from Bowman et al. (2022), an earlier study that found more optimistic results for simple human-plus-model protocols. It is a weak causal claim: that judges' own knowledge, which the models lacked, was one contributor to those positive results, not the sole cause. The earlier study's design makes the claim plausible from the start. Participants worked without internet access, so apart from the model's output their only resource was what they already knew. The model was a fairly weak language model that answered many questions wrongly, so a combined score above the model's own score had to come partly from what participants brought themselves. The reanalysis adds indirect evidence. On the factual MMLU task, in both the original and the replication data, correct answers came with significantly fewer conversation turns and resets than incorrect ones (for example, 3.6 versus 5.8 turns in the replication). On the QuALITY reading task there was no such pattern. This fits participants who already knew the answer answering quickly, and leaning on the model when they did not. The evidence is correlational and stops short of direct proof. Fewer turns could also reflect easier questions, where both humans and model do well, rather than humans knowing what the model did not. The reanalysis does not break down accuracy on the items the model got wrong by how much each participant knew beforehand, which would test the claim directly. The claim is therefore supported rather than verified: it is well motivated by the design and consistent with the interaction data, but the mechanism has not been isolated.
Claim entered the graph