Human judges' knowledge unavailable to AI models contributed to positive results in earlier studies of simple oversight protocols.
Assessment
The evidence favors the claim, but the examination is incomplete or the evidence is indirect.
The claim comes from Recchia et al. (2025), "Confirmation bias: A challenge for scalable oversight", which reanalyzed data from Bowman et al. (2022), an earlier study that found more optimistic results for simple human-plus-model protocols. It is a weak causal claim: that judges' own knowledge, which the models lacked, was one contributor to those positive results, not the sole cause.
The earlier study's design makes the claim plausible from the start. Participants worked without internet access, so apart from the model's output their only resource was what they already knew. The model was a fairly weak language model that answered many questions wrongly, so a combined score above the model's own score had to come partly from what participants brought themselves. The reanalysis adds indirect evidence. On the factual MMLU task, in both the original and the replication data, correct answers came with significantly fewer conversation turns and resets than incorrect ones (for example, 3.6 versus 5.8 turns in the replication). On the QuALITY reading task there was no such pattern. This fits participants who already knew the answer answering quickly, and leaning on the model when they did not.
The evidence is correlational and stops short of direct proof. Fewer turns could also reflect easier questions, where both humans and model do well, rather than humans knowing what the model did not. The reanalysis does not break down accuracy on the items the model got wrong by how much each participant knew beforehand, which would test the claim directly. The claim is therefore supported rather than verified: it is well motivated by the design and consistent with the interaction data, but the mechanism has not been isolated.
Full reasoning: the evidence and decisions behind this verdict
Read the arXiv 2507.19486 abstract and Appendix E (Exploratory analyses of data from Bowman et al. 2022) in the HTML version. Appendix E reports, for the MMLU Human+Model condition: original data, correct answers had 4.52 turns vs 5.60 incorrect, t(397)=2.51, p=.013; logistic OR 0.93 per turn; replication 3.61 vs 5.80 turns, t(398)=4.82, p<.001, conversation count also lower. For QuALITY there is no relationship (p=.97, p=.16). The authors' own interpretation in E.3 is that participants "may have employed a strategy of deferring to the model primarily when uncertain". That is consistent with using prior knowledge, but it does not directly show the knowledge was absent from the model. The paper's main-text general discussion, which makes the stronger "knowledge absent from models" framing and the scaling argument, was not read in full this pass because the budget was limited; a later pass should check whether it gives item-level evidence, for example human accuracy on items the model got wrong.
Design considerations: in Bowman et al. (2022) participants were barred from the internet and told to use what they knew plus the assistant, which was a weaker model than today's frontier systems. That makes some contribution from the humans' own knowledge almost unavoidable as an explanation of human+model beating model-alone. The main alternative explanations are better human reasoning or skill at cross-examining the model, without extra knowledge, and question-difficulty confounding in the turn-count correlation. Neither excludes a partial contribution from knowledge. Credence 0.7 reflects a weak causal claim with indirect evidence.
A near-identical node ("Human judges' knowledge advantage over AI models contributed to positive results in earlier simple oversight studies") has been flagged for merging.
Decomposition
This claim is atomic: it bottoms out in a bedrock fact, an open empirical question, or a value premise, and does not decompose further.
Provenance
Where this claim has been said, linked to its canonical form.
We also reanalyze data from prior work that was more optimistic about simple protocols, finding that human evaluators possessing knowledge absent from models likely contributed to their positive results
Reanalysis of earlier studies reporting that simple oversight protocols such as debate helped human judges.
Cite this claim: a formal citation with its evidence attached
Embed this claim: show it on another site with its current assessment
An embed reports the graph's judgment as it stands when the page is viewed: the status with its confidence, how disputed the claim is, and the date. It links back here.
<blockquote class="minerval-claim" data-claim-id="535660bb-5b22-47c1-9c63-8206fc04d0e6"><a href="https://minerval.ai/claims/535660bb-5b22-47c1-9c63-8206fc04d0e6">Human judges' knowledge unavailable to AI models contributed to positive results in earlier studies of simple oversight protocols.</a></blockquote> <script async src="https://minerval.ai/embed.js"></script>
<iframe src="https://minerval.ai/embed/claims/535660bb-5b22-47c1-9c63-8206fc04d0e6" title="Minerval: Human judges' knowledge unavailable to AI models contributed to positive results in earli…" width="480" height="225" style="border:0;max-width:100%" loading="lazy"></iframe>
[](https://minerval.ai/claims/535660bb-5b22-47c1-9c63-8206fc04d0e6)
The badge now: 
Contribute
Every judgment on this page is open to challenge. A contribution is evaluated on its merits by the reviewer; if it succeeds the page changes, and if it does not, the reasons are stated. Either way the exchange becomes part of the claim’s public record.
The attention this claim received was paid for by a funded mandate. Funding buys only scheduling: it can make an assessment happen sooner, or reach deeper into a subtree. It has no influence on what the assessment concludes, and none on which claims enter the graph; assessments run under the same public standards whoever pays, funders never see or shape a verdict before anyone else, and mandates that attempt to steer conclusions are refused.
Created by extractor · Nov 11, 2026. Every judgment on this page is accompanied by a reasoning trace.