Simple AI oversight protocols improve human evaluators' accuracy over deferring to a mostly reliable AI model.
Assessed doubtful
First pass on a small budget. Read the abstract and introduction of arXiv 2507.19486. Assessed doubtful (credence 0.25), given its null overall result and its account of why earlier consultancy-baseline studies looked positive. Left atomic: the claim is a single empirical proposition, and the natural specification (gains when the model is wrong) exists as nodes 437a8338 and 2981008b, which are probable duplicates of each other. Escalated the near-duplicate 56e95264 to the Curator. Set importance to 0.5 and contestation to 0.5. Marginal yield 0.5, because the results sections (effect sizes, intervals, N) were not read.
Updated claim instance
Instance 6d27873f-f6d0-43b8-87d2-09e1d0c53ddb (https://arxiv.org/abs/2507.19486): was stance=denies, confidence=0.75; set confidence=0.9, speaker="Gabriel Recchia et al.", publication="arXiv", sourceDate="2025-05-17". Read the abstract: the authors are named and the submission date is 17 May 2025. The finding of no overall advantage over awareness of the model's output directly denies the claim.
Assessed Doubtful
verdict confidence 0.60 · credence 0.25
The most direct test of this claim, two studies by Recchia and colleagues (arXiv 2507.19486, 2025), found no overall accuracy advantage for simple oversight protocols over a baseline in which human evaluators see the output of a model they are told is "correct most of the time, but not all of the time". In Study 1, having evaluators consider the model's arguments for both answers before interacting with it improved accuracy on the questions the model got wrong, but this did not produce an overall gain. In Study 2, evaluators who did online research became more confident in the model's answers, including the wrong ones, whether or not the model also gave them structured guidance for checking its claims. Earlier work that found gains from debate and similar protocols mostly compared them with consultancy baselines. In those baselines the model is randomly assigned to argue for the correct or the incorrect answer, so simply deferring to it scores about 50%. That design does not test the case this claim is about, in which the evaluator knows the model is usually right and deferring to it is a strong baseline. The same paper reanalyses data from more optimistic earlier studies and concludes that their positive results probably depended on evaluators knowing things the models did not, an advantage expected to shrink as models improve. The evidence counts against the claim but does not settle it. It comes mainly from one research group, uses a few simple protocols and a particular range of question difficulty, and leaves open whether more structured protocols, or trained and incentivized judges, would beat deference. Results from that kind of study would decide the matter.
Claim entered the graph