Auditing Games for Sandbagging

We found that the blue team could not reliably discriminate sandbaggers from benign models. Black-box approaches were defeated by effective imitation of a weaker model. Linear probes, a model-internals approach, showed more promise but their naive application was vulnerable to behaviours instilled by the red team. We also explored capability elicitation as a strategy for detecting sandbagging. Although Prompt-based elicitation was not reliable, training-based elicitation consistently elicited full performance from the sandbagging models, using only a single correct demonstration of the evaluation task. However the performance of benign models was sometimes also raised, so relying on elicitation as a detection strategy was prone to false-positives.

We found that the blue team could not reliably discriminate sandbaggers from benign models. ... training-based elicitation consistently elicited full performance from the sandbagging models, using only a single correct demonstration of the evaluation task. However the performance of benign models was sometimes also raised, so relying on elicitation as a detection strategy was prone to false-positives. In the short-term, we recommend developers remove potential sandbagging using on-distribution training for elicitation.

The red team successfully instilled behaviours into their models which induced false positives in Baobab and Oak. These models both have distractor behaviours which the red team hypothesised would cause false positives in probes: over-using double negatives, and repeating multiple choice question options.

Fuzzing can sometimes identify tasks where models can be made to perform better, but is unreliable and does not distinguish sandbagging from benign underelicitation.

The red team was successful at creating a sandbagging model organism (Cedar) on which the blue team could not find any examples of suspicious behaviours.