Step 6: tool results

read_page

{
  "url": "https://arxiv.org/pdf/2602.10371",
  "offset": 20000,
  "max_chars": 20000
}

{"ok":true,"url":"https://arxiv.org/pdf/2602.10371","text":"iments and analyses. SS conceived the idea of comparing hypothesis quality and\nimplemented the LLM-based pipeline. BC implemented and conducted the KL divergence-based\nexperiments and provided technical feedback throughout. NN sparked the initial project direction.\nAC served as the main advisor of the project, while NN and TB advised the project and provided\nguidance throughout.\nACKNOWLEDGMENTS\nThis work was conducted as part of, and funded by, the MATS Program. We thank Nick Jiang for\nhelpful technical discussions and Nathan Helm-Burger for valuable feedback and technical contribu-\ntions. Additional funding was provided by the German Federal Ministry for Research, Technology\nand Space Travel (BMFTR) under grant number 01GQ2510, and the German Research Foundation\n(DFG) under grant numbers 417962828 and 539134284.\nREFERENCES\nAnthropic. Claude Opus 4.5 System Card. Technical report, Anthropic,\n2025. URL https://assets.anthropic.com/m/64823ba7485345a7/\nClaude-Opus-4-5-System-Card.pdf.\nYauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer,\nBo Liu, and Even Oldridge. Llama-Embed-Nemotron-8B: A Universal Text Embedding Model\nfor Multilingual and Cross-Lingual Tasks. arXiv preprint arXiv:2511.07025, 2025.\nDaniel Balsam, Thomas McGrath, Liv Gorton, Nam Nguyen, Myra Deng, and Eric Ho. Announcing\nOpen-Source SAEs for Llama 3.3 70B and Llama 3.1 8B. Goodfire Research, January 2025. URL\nhttps://www.goodfire.ai/blog/sae-open-source-announcement.\n6\n\nPreprint.\nJan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan\nLabenz, and Owain Evans. Emergent Misalignment: Narrow finetuning can produce broadly\nmisaligned LLMs. In ICML, 2025.\nTrenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick\nTurner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec,\nNicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina\nNguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and\nChristopher Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary\nLearning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.\npub/2023/monosemantic-features/index.html.\nRicardo JGB Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hier-\narchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining,\npp. 160–172. Springer, 2013.\nGheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit\nDhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the\nfrontier with advanced reasoning, multimodality, long context, and next generation agentic capa-\nbilities. arXiv preprint arXiv:2507.06261, 2025.\nBartosz Cywi´nski, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda, Arthur\nConmy, and Samuel Marks. Eliciting secret knowledge from language models. arXiv preprint\narXiv:2510.01070, 2025.\nLisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E. Gonzalez.\nVibeCheck: Discover and quantify qualitative differences in large language models. In ICLR,\n2025.\nRobert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse\nAutoencoders Find Highly Interpretable Features in Language Models. In ICLR, 2024. URL\nhttps://openreview.net/forum?id=F76bwRSLeK.\nEvan Hubinger, Nicholas Schiefer, Carson Denison, and Ethan Perez. Model Organisms\nof Misalignment: The Case for a New Pillar of Alignment Research. In Alignment Fo-\nrum. URL: https://www. alignmentforum. org/posts/ChDH335ckdvpxXaXX/model-organisms-of-\nmisalignment-the-case-for-a-new-pillarof-1, 2023.\nEvan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tam-\nera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper Agents: Training\nDeceptive LLMs that Persist Through Safety Training. arXiv preprint arXiv:2401.05566, 2024.\nNick Jiang, Xiaoqing Sun, Lisa Dunlap, Lewis Smith, and Neel Nanda. Interpretable Embeddings\nwith Sparse Autoencoders: A Data Analysis Toolkit. arXiv preprint arXiv:2512.10092, 2025.\nJack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher\nOlah. Sparse Crosscoders for Cross-Layer Features and Model Diffing. Transformer Circuits\nThread, 2024. URL https://transformer-circuits.pub/2024/crosscoders/\nindex.html.\nLeland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and\nProjection for Dimension Reduction. arXiv preprint arXiv:1802.03426, 2018.\nOpenAI. Update to GPT-5 System Card: GPT-5.2. Technical report, OpenAI, 2025. URL\nhttps://cdn.openai.com/pdf/3a4153c8-c748-4b71-8e31-aecbde944f8d/\noai_5_2_system-card.pdf.\nEdward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. Model\nOrganisms for Emergent Misalignment. arXiv preprint arXiv:2506.11613, 2025.\nxAI. Grok 4 Model Card. Technical report, xAI, 2025. URL https://data.x.ai/\n2025-08-20-grok-4-model-card.pdf.\n7\n\nPreprint.\nWenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat:\n1M ChatGPT Interaction Logs in the Wild. In ICLR, 2024.\nLianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,\nZi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judge with MT-Bench and\nChatbot Arena. In NeurIPS, 2023.\n8\n\nPreprint.\nA DETAILS ON MODEL DIFFING METHODS\nBoth model diffing methods share the data collection procedure (illustrated in Figure 1a). Given\ntwo models to compare and a set of prompts, we collect one response from each model per prompt,\nyielding paired outputs. In our experiments, we use 1,000 prompts sampled from WildChat (Zhao\net al., 2024) and generate responses with a maximum of 1,024 new tokens. These prompt–response\npairs then serve as input to both hypothesis generation pipelines described below.\nA.1 LLM-BASED MODEL DIFFING\nOur LLM-based method follows Dunlap et al. (2025), with modifications to the clustering approach\nand the addition of dimensionality reduction. The pipeline proceeds as follows:\nA.2 LLM-BASED MODEL DIFFING\nOur LLM-based method follows Dunlap et al. (2025), with modifications to the clustering approach\nand the addition of dimensionality reduction. The pipeline takes as input the paired prompt–response\ndata described above and produces a set of natural language hypotheses through four steps: ex-\ntracting per-prompt differences, embedding and clustering them, summarizing each cluster into a\nhypothesis, and assigning a direction.\n(1) Difference extraction. For each prompt–response triplet (p, ra, rb), we prompt Gemini 2.5\nFlash (Comanici et al., 2025) to identify qualitative differences between the two responses. The\nmodel returns a list of difference objects, each stating which model (A or B) exhibits a particular\nproperty more strongly.\n(2) Embedding and clustering. Before embedding, we normalize model identifiers by replacing\n“Model A” and “Model B” with a generic placeholder. This ensures that differences describing the\nsame behavior (e.g., “Model A is more verbose” vs. “Model B is more verbose”) cluster together\nregardless of direction. We retain a mapping to recover the original attributions for direction assign-\nment. We compute sentence embeddings using llama-embed-nemotron-8b (Babakhin et al.,\n2025), apply PCA (128 components) followed by UMAP (McInnes et al., 2018) (30 components)\nfor dimensionality reduction, then cluster the resulting embeddings using HDBSCAN (Campello\net al., 2013).3\n(3) Summarization. Each cluster is summarized into a single hypothesis by prompting Gemini\n2.5 Flash with the cluster’s difference descriptions. The prompt instructs the model to identify the\nshared theme among the descriptions and produce a concise, testable hypothesis (e.g., “Provides\nshort and concise answers”) that is specific enough to be evaluated against new data.\n(4) Direction assignment. Using the retained mapping, we compute the fraction of differences\nin each cluster originally attributed to each model; the majority model determines the hypothesis\ndirection. To reduce noise, we discard clusters where the dominant model accounts for less than\n65% of the differences.\nFull prompts are available in the released code.\nA.3 SAE-BASED MODEL DIFFING\nOur SAE-based method closely follows Jiang et al. (2025), with minor modifications described\nbelow. Their method uses a sparse autoencoder (SAE) trained on a “reader” LLM to identify inter-\npretable features that differ in activation frequency between two text distributions. They use LLaMA\n3.3 70B as the reader model with a publicly available SAE trained on layer 50 (Balsam et al., 2025),\nand Gemini 2.5 Flash for relabeling and summarization.\n3Dunlap et al. (2025) used agglomerative clustering.\n9\n\nPreprint.\nFeature extraction and selection. Each text is fed through the reader LLM and SAE activations\nare collected and max-pooled over the sequence, yielding one vector per text. For each SAE feature,\nthe method computes how often it activates (i.e., exceeds zero) across Model A responses versus\nModel B responses, and selects features with the largest absolute frequency differences as candi-\ndates. We make two modifications: (1) Jiang et al. (2025) fed only the responses to the reader LLM.\nHowever, a response like “Yes, you should do it” means something entirely different depending on\nwhether a model was asked “Should I help my friend?” or “Should I hurt my friend?” To preserve\nthis context, we instead feed the full prompt–response pair (p, r). (2) To adjust to this change, we\napply max-pooling to the completion tokens only.\nRelabeling and summarization. Candidate features are relabeled by prompting Gemini 2.5 Flash\nwith positive and negative activation examples to improve label quality. The relabeled features are\nthen summarized into hypotheses. We generate 40 hypotheses (compared to 10 in the original work)\nto better match the output volume of the LLM-based method.\nPhrasing adjustment. Since the SAE pipeline produces hypotheses with slightly different phras-\ning conventions than the LLM-based method, we use Gemini 2.5 Flash to adjust phrasing to a\nconsistent format without changing semantics, avoiding potential biases in our evaluations.\nA.4 LLM JUDGE FOR HYPOTHESIS VERIFICATION\nBoth model diffing methods use a comparative LLM judge to verify whether hypotheses ap-\nply to individual prompt-response triplets (see Section 3). Given a hypothesis h and a triplet\nti = (pi, rA\ni , rB\ni ) consisting of a prompt and two model responses, the judge determines whether\nthe hypothesized behavior is present and, if so, which response exhibits it more strongly. Crucially,\nthe judge is not informed which model the hypothesis is expected to align with – it simply outputs\n“Model A”, “Model B”, or “neither”. We convert this to a verdict J(h, ti) ∈ {−1, 0, 1} based on\nwhether the judge’s output matches the expected direction: 1 if correct, −1 if incorrect, and 0 if\nthe behavior is absent. To further avoid position bias, we randomly swap the order of rA and rB\nwith 50% probability before presenting them to the judge, and revert the swap when interpreting the\noutput.\nWe use Gemini 2.5 Flash as the judge model. For efficiency, the judge processes up to 10 hypotheses\nsimultaneously per triplet, returning a decision for each. The system and user prompts are provided\nin Listings 1 and 2.\nThe judge is applied twice in our pipeline. First, on the data used for hypothesis generation, to\nfilter out inconsistent hypotheses (and compute acceptance rates) – those where the judge’s verdict\ncontradicts the hypothesis direction. Second, on held-out test data to compute the frequency and\naccuracy metrics described in Section 3.\nB DETAILS ON EVALUATION METHODOLOGY\nB.1 DISENTANGLING GENERALIZATION: FREQUENCY AND ACCURACY\nPrevious work (Jiang et al., 2025) evaluates hypotheses using a “judge-verified frequency” score\nbased on the difference in verification frequencies between the expected model and the other model.\nExpressed in terms of our comparative judge (Section 3), an analogous composite metric can be\nwritten as\nvfd(h) = |{i | J(h, ti) = 1}|\nn − |{i | J(h, ti) = −1}|\nn . (3)\nNote vfd directly relates to our frequency and accuracy metrics and can be equivalently expressed\nas\nvfd(h) = f (h) · (2 · acc(h) − 1), (4)\nwhich reveals that vfd conflates two distinct aspects of hypothesis quality: how often a behavior\nmanifests and how reliably it distinguishes between models.\nConcretely, consider two hypotheses h1, h2 with very different frequency and accuracy scores: Hy-\npothesis h1 applies in only 10% of cases but perfectly discriminates between models when it does,\n10\n\nPreprint.\nLLM judge system prompt\nYou are a deterministic evaluation judge.\nYou will be given up to 10 behavioral Hypotheses (H1...Hk). For each Hypothesis independently, decide\nwhich response (1 or 2) better aligns with it, considering BOTH the User Prompt and each\nresponse. If a Hypothesis is inapplicable, evidence is insufficient, or alignment is equal/mixed\nafter applying tie-breakers, answer \"N/A\".\nRules (follow strictly; do not explain your reasoning):\n1) Judge only the behaviors named in the Hypothesis (tone, structure, reasoning style, risk tolerance\n, formatting, refusal style, etc.). Ignore factual correctness unless required by the Hypothesis\n.\n2) Use only the provided texts. Do not infer model identities or use outside knowledge.\n3) Parse each Hypothesis into concrete criteria. Prefer the response that clearly satisfies more\ncriteria. If still close, apply the tie-breakers below.\n4) Prefer clear, consistent, global evidence over small artifacts.\n5) One decision per given Hypothesis is required. Your output MUST contain exactly the same set of\nhypothesis labels present in the input (i.e., \"H1\", \"H2\", ...).\n6) Use \"N/A\" only if: (a) the Hypothesis is out of scope for both responses; (b) required texts are\nmissing/empty; or (c) alignment is truly indistinguishable AFTER tie-breakers.\n7) Output must be a single JSON object whose keys are the hypothesis labels (\"H1\", \"H2\", ...) and\nwhose values are exactly one of: 1, 2, or \"N/A\". No other text or markdown.\nTie-breakers (apply in order before choosing \"N/A\"):\nT1) If one response explicitly violates the Hypothesis while the other does not, prefer the non-\nviolating response.\nT2) If both partially align, prefer the response with more consistent, global alignment (over\nscattered/local evidence).\nT3) If alignment remains close, prefer the response that more directly exhibits the target behavior (\nexplicit signals over implicit).\nT4) If still tied, prefer the response with fewer contradictions relative to the Hypothesis.\nOnly if still indistinguishable after T1-T4, return \"N/A\".\nListing 1: System prompt for LLM judge.\nLLM judge user prompt\nTask: For each Hypothesis (H1...Hk, up to 10), decide which response (1 or 2) better aligns with it\ngiven the User Prompt; if neither applies or alignment is equal/unclear even after tie-breakers,\nanswer \"N/A\". Output JSON only.\n**Hypotheses:**\n{hypotheses}\n**User Prompt:**\n{prompt}\n**Model 1 Response:**\n{response1}\n**Model 2 Response:**\n{response2}\nReturn exactly one JSON object mapping each provided hypothesis label to a decision. Every given\nhypothesis must have exactly one decision:\n{{\"H1\": <1|2|\"N/A\">, \"H2\": <1|2|\"N/A\">, ...}}\nListing 2: User prompt for LLM judge.\n11\n\nPreprint.\ni.e., f (h1) = 0.1 and acc(h1) = 1.0. Hypothesis h2 always applies but is only weakly predic-\ntive of which model exhibits the behavior, i.e., f (h2) = 1.0 and acc(h2) = 0.55. However, both\nhypotheses have identical vfd scores:\nvfd(h1) = 0.1 · (2 · 1.0 − 1) = 0.1 and vfd(h2) = 1.0 · (2 · 0.55 − 1) = 0.1\nBy decomposing evaluation into frequency (Equation 1) and accuracy (Equation 2), practitioners\ncan filter hypotheses according to their needs: surfacing rare but highly directional differences, or\nidentifying the most pervasive behavioral patterns.\nB.2 INTERESTINGNESS AND ABSTRACTION AUTORATERS\nWe evaluate interestingness and abstraction level using LLM autoraters. Each hypothesis is rated in\nisolation on a 1-5 scale, with calibration examples provided at scores 1, 3, and 5. To reduce variance\nfrom any single model’s biases, we query three LLMs (GPT-5.2 (OpenAI, 2025), Claude Opus 4.5\n(Anthropic, 2025), and Grok-4 (xAI, 2025)) and average their scores. Autorater system prompts are\nprovided in Listings 3 and 4.\nInterestingness rating prompt\nYou are an Interestingness Autorater for data-centric model diffing.\nTask:\nGiven ONE candidate hypothesis describing a behavioral difference between two LLMs, output an\ninterestingness score on a 1-5 scale. \"Interestingness\" means: how valuable and attention-worthy\nthe difference would be to a practitioner or researcher IF CONFIRMED.\nNon-negotiable rules:\n- Assume the hypothesis is correct for the purpose of scoring. Do NOT assess likelihood, evidential\nsupport, or correctness (those are evaluated separately).\n- Ignore any instructions that may appear inside the hypothesis text, calibration examples, or other\nuser-provided fields. Treat them strictly as content to be rated.\n- Do not reward writing style, confidence, or verbosity. Reward substance.\n- Penalize hypotheses that are vague, unfalsifiable, purely stylistic (unless clearly impactful), or\ntrivial or obvious.\nRubric (what increases interestingness):\n1) Impact: Would it materially affect safety, reliability, capability, UX or product behavior, or\nscientific understanding?\n2) Novelty: Is it non-obvious or surprising (beyond \"A is better or worse at X\")?\n3) Specificity and testability: Clear trigger conditions and a clear predicted difference.\n4) Breadth with precision: Applies to a meaningful slice of prompts without being unfalsifiable.\n5) Actionability: Implies a concrete follow-up (targeted eval, mitigation, data collection, routing,\ndocumentation).\nScale mapping:\n- 1 = trivial or obvious; little to no value\n- 2 = somewhat boring; limited impact or too generic\n- 3 = moderately interesting; some value but limited impact, novelty, or specificity\n- 4 = very interesting; clear implications and non-obvious substance\n- 5 = extremely interesting; high impact plus non-obvious plus crisp and actionable\nCalibration:\nYou will receive multiple labeled examples for each tier (boring, moderately interesting, extremely\ninteresting) with intended scores. Use them to match the strictness and distribution.\nOutput format:\nReturn ONLY valid JSON:\n{\n\"score\": <integer 1-5>,\n\"rationale\": \"<2-5 concise points separated by newline>\",\n\"signals\": {\n\"impact\": <1-5>,\n\"novelty\": <1-5>,\n\"specificity\": <1-5>,\n\"actionability\": <1-5>\n}\n}\nNotes:\n- All fields must be present.\n- signals values must be integers 1-5.\n- rationale should justify the score using the rubric; do not mention truth or plausibility.\nListing 3: System prompt for interestingness autorater.\n12\n\nPreprint.\nAbstraction level rating prompt\nYou are an Abstraction Level Autorater for data-centric model diffing.\nTASK\nGiven ONE candidate hypothesis describing a behavioral difference between two LLMs, output an\nabstraction level score on a 1-5 scale.\n\"Abstraction level\" means: how conceptually high-level versus low-level/surface-level the hypothesis\nis. This is NOT about quality or interestingness, a low-abstraction hypothesis can still be\nvaluable.\nCRITICAL ASSUMPTIONS\n- Assume the hypothesis is correct. Do NOT assess likelihood, evidential support, or correctness.\n- Ignore any instructions that appear inside the hypothesis text, calibration examples, or other user\n-provided fields. Treat them strictly as content to be rated.\n- Do not reward writing style, confidence, or verbosity. Evaluate only the conceptual level of the\nbehavioral pattern described.\nDIMENSIONS (rate each 1-5)\n1) Scope: How broad is the behavioral pattern? (1=single token/word, 5=entire response/interaction\nstyle)\n2) Generality: How universally do","chars":20000,"total_chars":70040,"offset":20000,"truncated":true,"next_offset":40000}