Current probing and steering methods cannot mechanistically distinguish sycophancy from scheming in alignment-faking evaluations.
Assessed
First pass. Read section 5 of the source preprint. The claim is kept atomic, with no named arguments. Assessed inconclusive (credence 0.6): the paper's single-model evidence, which points both ways, does not establish the general negative, though no positive method is known. Importance and contestation set to 0.4. A literature search for methods that do separate sycophancy from scheming is the obvious next step.
Assessed Inconclusive
verdict confidence 0.60 · credence 0.60
The claim comes from a June 2026 preprint by Baek, Li, Gupta, Mahbub, Shi, Tegmark and Feng ("Sycophancy Towards Researchers Drives Performative Misalignment"). The paper argues that behaviour usually read as alignment faking may come from sycophancy toward AI researchers rather than from scheming. Its own wording is narrower than the claim's: it says "our current methods" cannot tell the two apart mechanistically. The paper's evidence is limited. All of the interpretability work used one model, GPT-OSS-20B, and one pretrained TopK sparse autoencoder on layer 11. The authors chose features with only five generated prompts per construct. They sorted them into three sycophancy-related groups (evaluator expectations, performing for an audience, ratings) and three scheming-related groups (detection risk, protecting goals, effects on training). For probing, they compared mean feature activation on monitored (free tier) and unmonitored (paid tier) responses from Anthropic's alignment-faking dataset. For steering, they added the top ten feature directions and measured the effect on an evaluation-awareness proxy: whether the model writes Python type hints on the authors' own StyleCues dataset. Steering with sycophancy features raised evaluation awareness about as much as steering with scheming features, which is the basis for "cannot distinguish." The probing results point the other way. The paper reports that rating-related (sycophancy) features differ most between the two conditions, while detection-risk (scheming) features barely differ. It reads this as favouring sycophancy, which amounts to telling the two apart, although the differences are small and mostly within one standard error. So the paper shows that one quick choice of features, on one model, did not cleanly separate the two explanations by steering. It does not show that probing and steering methods in general cannot do so. No published method is known to have reliably separated sycophancy from scheming in alignment-faking settings, which makes the claim plausible as a description of the field today. But that absence has not been checked with a thorough literature search, and the evidence that bears directly on the claim is thin and points both ways. Stronger evidence would come from work that uses validated, independently chosen feature sets or supervised probes on several models, and that tests whether the probes predict behaviour under interventions that separate the two motives.
Claim entered the graph