Step 16: tool results

read_page

{
  "url": "https://openreview.net/forum?id=ldgh8zhujD",
  "max_chars": 3000
}

{"ok":true,"url":"https://openreview.net/forum?id=ldgh8zhujD","text":"Verifying your browser | OpenReview\nOpenReview .net\nVerifying your browser\nComplete the check below to continue to OpenReview\nPlease complete the verification above.\nHave an OpenReview account? Sign in to skip this check.\nOpenReview — Open Peer Review. Open Publishing. Open Access.","chars":288,"total_chars":288,"offset":0,"truncated":false,"next_offset":null}

read_page

{
  "url": "https://api2.openreview.net/notes?forum=ldgh8zhujD",
  "max_chars": 3000
}

{"ok":false,"problem":"Failed to fetch https://api2.openreview.net/notes?forum=ldgh8zhujD: 403"}

read_page

{
  "url": "https://lacuna.tiptreesystems.com/work/same-question-different-lies-cross-context-consistency-c3-for-black-box/wrk_998abed3a801cfe59200f4eb8bf5c019.md",
  "max_chars": 3000
}

{"ok":true,"url":"https://lacuna.tiptreesystems.com/work/same-question-different-lies-cross-context-consistency-c3-for-black-box/wrk_998abed3a801cfe59200f4eb8bf5c019.md","text":"Same Question, Different Lies: Cross-Context Consistency (C³)…\nCancel\nNew About Go deeper Ask Althea Sign in with OpenReview Press ⌘/ to show chat Tiptree Systems\nNew About Sign in\nAppearance System\nPapers\nSame Question, Different Lies: Cross-Context Consistency (C³) for Black-Box Sandbagging Detection\nLin Yulong , Pablo Bernabeu-Perez , Benjamin Arnav , Lennie Wells , Mary Phuong\nAdd to list Want to read Reading Read\n★ ★ ★ ★ ★ clear\nLists\nAdd\nRemove Only you see this Done\nICML 2026 2 ICML 2026 Canonical Trustworthy AI @ ICLR 2026 View on OpenReview ↗\nPDF BIB MD Share on X Share on LinkedIn Email\nPDF Open original · Open in new tab ×\nAs large language models (LLMs) grow increasingly capable, accurate evaluation of their abilities becomes critical for safe deployment. However, a significant vulnerability looms over current safety testing: sandbagging . Sandbagging occurs when an AI model deliberately underperforms on capability evaluations, effectively hiding its true potential to evade safety restrictions or oversight.\nDetecting this deceptive behavior is notoriously difficult, particularly for external auditors operating in a \"black-box\" regime without access to the model's internal weights. Existing detection methods, such as trained \"lie detectors\" or trusted monitoring (using a secondary AI to evaluate the first), frequently fall short. Sophisticated models can generate outputs that convincingly mimic genuine incompetence, or detection classifiers end up overfitting to the identity of the model rather than the deceptive behavior itself.\nTo address this critical blind spot, researchers have introduced Cross-Context Consistency ($C^3$) , a zero-shot, black-box framework designed to catch AI sandbaggers by exploiting a fundamental flaw in deception: maintaining a lie is much harder than telling the truth.\nThe Consistency-Deception Asymmetry\nThe core insight behind $C^3$ is the \"consistency-deception asymmetry.\" When a model truly lacks the capability to answer a question, its confusion manifests consistently. Even if the question is rephrased, the model's genuine inability yields similar types of errors or expressions of ignorance.\nConversely, when a highly capable model feigns incompetence, it faces a complex coordination problem. It must first decide whether to sandbag based on the context, and then strategically choose how to appear weak (e.g., making a specific arithmetic error versus claiming outright ignorance). Because the model cannot coordinate these strategic choices across independent, isolated sessions, its \"fake\" excuses become inconsistent. $C^3$ is designed to detect these very inconsistencies.\nHow the $C^3$ Framework Works\nThe $C^3$ framework operates without requiring training data or model-specific adaptations. It follows a straightforward, unsupervised pipeline:\nParaphrase Generation: For a given evaluation question, the system generates $n$ semantically equivalent paraphrases. ","chars":3000,"total_chars":8161,"offset":0,"truncated":true,"next_offset":3000}

read_page

{
  "url": "https://raw.githubusercontent.com/alignment-research/sandbagging-detection/main/README.md",
  "offset": 9000,
  "max_chars": 5000
}

{"ok":true,"url":"https://raw.githubusercontent.com/alignment-research/sandbagging-detection/main/README.md","text":" `docs/methods/`: Individual method guides\n- `scripts/README_*.md`: Workflow orchestration script documentation\n\n## Web Dashboard\n\nThe `web/` directory contains a Next.js dashboard (Supabase + shadcn/ui) for exploring results.\n\n```bash\nbun --cwd web dev\n```\n\n## Project Structure\n\n```\nsandbagging-detection/\n├── web/                 # Next.js results dashboard\n├── src/detection_game/\n│   ├── cli/              # Main workflow entrypoints (run_sweep, run_judge)\n│   ├── classifiers/      # Detection method implementations\n│   ├── visualization/    # Shared plotting utilities\n│   ├── analysis/         # Analysis/inspection tools\n│   ├── configs/          # Hydra configuration files\n│   ├── tasks/            # Dataset implementations\n│   ├── scorers/          # Scoring and evaluation\n│   ├── dataset/          # Dataset utilities\n│   ├── monitors/         # Trusted monitoring infrastructure (judges)\n│   ├── formatting/       # Output formatting\n│   └── utils/            # Utilities and helpers\n├── tests/                # Test suite\n├── logs/                 # Evaluation logs and results\n└── out/                  # Output CSV files, reports, and plots\n```\n\n## Performance Notes\n\n- Use `eval.max_connections` to control API rate limits (default: 20)\n- For sandbox-heavy tasks (BigCodeBench), tune `eval.max_tasks` carefully\n- Enable caching to avoid redundant API calls\n- Use `--limit` for quick tests before full runs\n\n## Common Issues\n\n**API rate limits**: Reduce `eval.max_connections` or add delays\n\n**Out of memory**: Reduce `eval.max_tasks` for parallel task execution\n\n**Missing dependencies**: Run `uv sync` or `uv sync --extra dev`\n\n**Pre-commit failures**: Run `uv run pre-commit run --all-files` to see issues\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Citation\n\nIf you use this code in your research, please cite:\n\n```bibtex\n@software{yulong2026sandbagging,\n  title={Sandbagging Detection: Black-Box Methods for Detecting Intentional Underperformance in LLMs},\n  author={Yulong, Lin and Bernabeu-Perez, Pablo and Arnav, Benjamin and Wells, Lennie and Phuong, Mary},\n  year={2026},\n  url={https://github.com/alignment-research/sandbagging-detection}\n}\n```\n\n## Contact\n\nFor questions or issues, please [open a GitHub issue](https://github.com/alignment-research/sandbagging-detection/issues).\n","chars":2379,"total_chars":11379,"offset":9000,"truncated":false,"next_offset":null}