Step 2: tool results

get_claim

{
  "include": [
    "arguments",
    "subclaims",
    "dependents"
  ],
  "claim_id": "9fbf1b17-4bf8-42bc-a5e1-420054bf7463"
}

{"claim":{"id":"9fbf1b17-4bf8-42bc-a5e1-420054bf7463","kind":"claim","canonical_form":"Sparse autoencoder probes do not consistently outperform baseline probes on language-model activation classification tasks.","claim_type":"empirical_derived","state":"active","decomposition_status":"complete","importance":0.6,"contestation":0.55,"domains":[],"tags":[{"slug":"mechanistic-interpretability","name":"Mechanistic Interpretability"},{"slug":"large-language-models","name":"Large language models"}],"steward_state":"running","stewarded_at":"2026-11-14T20:39:59.901Z","canonical_direction_note":"This affirmative form states the thesis under dispute: whether sparse-autoencoder probes offer a consistent advantage over standard baselines in activation-based classification.","created_at":"2026-11-13T00:50:29.013Z","updated_at":"2026-11-14T20:39:56.822Z","page_url":"https://minerval.ai/claims/9fbf1b17-4bf8-42bc-a5e1-420054bf7463"},"assessment":null,"coverage":{"examined":false,"on":[],"note":"No judgment of this node is in force. That is not a finding either way."},"quantity":null,"threshold_of":null,"question":null,"formalization":null,"formalization_pending":null,"formalization_history":[],"verification":null,"lean_checks":[],"lean_checks_total":0,"waiting_messages":[{"trigger":"structure_and_assess","from":"the system","sender_node_id":null,"sent_at":"2026-11-13T00:50:29.161Z","paid":false,"summary":"New claim onboarded. Give it the structure and assessment its importance warrants; a minor or settled claim may need no decomposition."},{"trigger":"mandate_plan","from":"a mandate's plan","sender_node_id":null,"sent_at":"2026-11-14T20:39:56.857Z","paid":true,"summary":"First assessment. Open the primary probing studies linked to this node and its parent question (among them arXiv 2502.16681, 2505.20063, 2605.31183, 2602.14111 and the 'Negative results for SAEs on do…"}],"requests":[],"requests_total":0,"subclaim_count":0,"instance_count":4,"source_map":null,"subclaims":[],"subclaims_assessed":0,"arguments":[],"dependents":[{"id":"cb2d56c9-fcce-47d3-b2b3-5127f10f3293","text":"Under what conditions, if any, do sparse-autoencoder and other dictionary-learning features outperform linear probes and difference-of-means steering on labeled language-model tasks?","claim_type":"empirical_derived","node_kind":"question","relation_type":"answers","reasoning":"The null answer for detection and probing: across the regimes where an advantage was expected (few labels, class imbalance, label noise, covariate shift), probes on SAE latents show no consistent advantage over tuned probes on raw activations. It is the headline of the largest head-to-head probing study and is the answer most often cited.","importance":0.4,"assessment_status":null,"assessment_confidence":null,"assessment_credence":null,"bounty_micro_usd":null,"checked":null,"formal":false}],"dependents_total":1,"unplaced_argument_passages":[{"instance_id":"d28607d2-b9e2-439c-99ae-e9dda9c0b5dd","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"The quiver of arrows approach is designed as a robustness\ncheck to fairly evaluate the benefit of adding SAE probes to\na practioner’s toolkit.","context":"Method: choose best probe by validation AUC with and without SAE probes, compare test AUC.","stance":"advances","direction":"for"},{"instance_id":"4a0b790c-b632-4c69-9d8b-4311afe086f1","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"At all pa-\nrameter values in each regime, SAEs show no meaningful\nimprovement over baselines.","context":"Results for data scarcity, class imbalance and label noise regimes.","stance":"advances","direction":"for"},{"instance_id":"d43ecd5d-4d2b-434f-958b-b3efa0cf70b4","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"Without using the quiver of arrows approach, we still find SAEs underperform baselines.","context":"Appendix C.4, showing the negative result is not an artifact of the selection procedure.","stance":"advances","direction":"for"},{"instance_id":"93ae7c6f-0a23-4cf9-a3f8-8025b43a9cd5","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"Thus, latent 122774 does not generalize OOD, which\nexplains why the SAE probe also failed to generalize.","context":"Explaining the failure: a 'living room' latent fires on English but not French.","stance":"advances","direction":"for"},{"instance_id":"4bb736f5-b249-418c-a54d-06efa5cc27e0","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"Crucially, they simi-\nlarly max-pool each model dimension across prompt tokens\nfor their comparative baseline, even though activations are\nunlikely to have privileged dimensions","context":"Criticizing the baseline used by Bricken et al. (2024a) for multi-token SAE probing.","stance":"criticizes","direction":null},{"instance_id":"967e1659-1942-47f6-871f-6901e8cd5215","source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing"},"verbatim_text":"We find that when compared to the last-token\nbaseline, max-pooled SAE probes win 19.6% of the time, a\nconsiderable improvement over win rate of last-token SAE\nprobes (2.2%, see Figure 11). However, when implementing\nattention-pooled baselines and using the quiver of arrows\napproach to select between pooled and last-token strategies\nfor SAE probes and baselines, the SAE probe win rate drops\nmore than 50% to 8.7% (see Figure 11).","context":"Multi-token experiment on 60 datasets.","stance":"advances","direction":"for"},{"instance_id":"783fe2e1-47da-4cfe-ad8c-6bea5f0cda10","source":{"id":"cfaeaedc-6d7d-49fd-9a30-9f4874fcbb27","url":"https://arxiv.org/pdf/2506.23845","title":"Position: Use Sparse Autoencoders to Discover Unknowns"},"verbatim_text":"A reconstruction\nencodes strictly less information about a token than the orig-\ninal LM representation. It follows that, compared to the\noriginal representation, there is less information available\nin the SAE representation to predict the presence of a con-\ncept.","context":"Proposed explanation of the negative probing results.","stance":"advances","direction":"for"}],"unplaced_argument_passages_total":7}