Step 8: tool results

provenance_read_source

{
  "offset": 24000,
  "max_chars": 12000,
  "source_id": "b45014e2-63ec-41af-8a6f-59db01b1e0fc"
}

{"success":true,"source":{"id":"b45014e2-63ec-41af-8a6f-59db01b1e0fc","url":"https://arxiv.org/pdf/2502.16681","title":"Are Sparse Autoencoders Useful? A Case Study in Sparse Probing","source_type":"preprint"},"origin":"stored","total_chars":97632,"offset":24000,"content":"le ranking latent 51330, which identifies “objects\nand materials related to scientific experiments,” last.\nWe then construct a probe with the top k latents using o1’s\nrelevance rank for k ∈ [1, 8]. If there are spurious latents\nin the k = 8 probe, we expect a probe with k < 8 to have\nbetter OOD generalization. We visualize each task’s per-\nformance by k in Figure 20, Appendix G.1. For two of\nthe datasets, 66 living-room and 90 glue qnli, we see that\npruning works, with a 0.024 and 0.052 increase in AUC\nbetween the k = 8 and k = 1 probe for each dataset.\nHowever, for 7 hist fig ispolitician, OOD test AUC in-\ncreases by 0.077 after pruning. This indicates that the task\n7 hist fig ispolitician requires k = 8 latents to represent.\nOur preliminary results show that pruning helps SAE probes\ngeneralize. However, the drop between in-distribution (ID)\nand OOD performance is much larger than the modest im-\nprovement from pruning. This indicates that spurious cor-\nrelations are not the primary reason SAE probes fail to\ngeneralize. Thus, we consider the second hypothesis - that\nthe underlying SAE latents do not generalize well OOD.\nOn the task 66 living-room, latent 122774 has an ID test\nAUC of 0.99 while only having an OOD test AUC of\n0.64. Since the OOD transformation involves translating\nthe prompts to French, we hypothesize that this latent is\nactive on the phrase “living room” in English but not other\nlanguages. We use GPT-4o to translate “living room” to 15\nlanguages, and in Figure 8 we indeed observe that latent\n122774 is more active on the English translation than all\nother languages, and it is not active on the French translation\nat all. Thus, latent 122774 does not generalize OOD, which\nexplains why the SAE probe also failed to generalize.\n4.2. Latent Interpretability\nWe next generate autointerp descriptions for the most de-\nterminative latent for each dataset, allowing us to iden-\ntify interesting categories of latents. Specifically, we con-\n6\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse ProbingEnglish\nSpanish\nFrench\nGerman\nItalian\nPortuguese\nRussian\nChinese\nJapanese\nKorean\nArabic\nHindi\nBengali\nDutch\nTurkish\nLanguage\n0\n20\n40\nLatent Value Latent 122774 on \"Living Room\" in Different Languages\nFigure 8. Latent 122774 is most active on the English translation\nof “living room,” and does not fire on French.\nsider each of the top 128 latents for each dataset for the\nwidth = 131k, L0 = 114 SAE, and evaluate each latent’s\nk = 1 SAE probe in standard settings. We then generate an\nautointerp explanation with GPT-4o for the best latent.\nTable 5 (Appendix G.2) contains a breakdown of a selec-\ntion of interesting top latents that we find. We see that\nsome latents’ descriptions fit their tasks, like latent 81210\nfor 5 hist fig ismale, which activates on “references to fe-\nmale individuals.” Another interesting category is incorrect\nautointerp labels (e.g. for 125 wold country Italy, latent\n50817 has 0.989 AUC, implying (correctly) that it fires on\nItalian concepts; however, the description reads “names of\nresearchers and scientific authors”). However, some latents\nappear to be spurious and unrelated to the semantics of their\ndataset. For example, 22 headline isobama, which targets\nheadlines published during the Obama administration, is\nclassified with 0.782 AUC by latent 10555, which activates\non “strings of numbers, often with mathematical notations.”\nNote that these findings may be possible using baseline clas-\nsifiers. For example, we could take a baseline classifier and\napply it to model hidden states on tokens from the Pile (Gao\net al., 2020) and then examine maximally activating exam-\nples. However, a practical advantage for SAEs is that the\ninfrastructure to perform autointerp is pre-existing through\nplatforms like Neuronpedia, and a theoretical advantage is\nthat the baseline classifier can only identify the single most\nrelevant coarse-grained feature, while the decomposabil-\nity of SAE probes into latents allows for identifying many\nindependent features of various importance.\n4.3. Detecting Dataset Quality Issues\nWe now investigate the top latents of two datasets,\n87 glue cola and 110 aimade humangpt3. We find that\nalthough the latents identify errors in each dataset, baseline\nmethods are also able to identify these dataset errors.\n4.3.1. GLUE COLA\nWe first examine 87 glue cola, an established linguistic\nacceptability dataset. CoLA prompts are standard EnglishCoLA Labels Clean Labels Mislabeled CoLA\n0.0\n0.5\n1.0\nTest Accuracy 1.00\n0.81 0.77 0.72 0.78 0.80 0.80 0.78\n0.00\n0.46\n0.56 0.63\nOriginal Labels\nBaseline\nSAE (k=128)\nLatent 369585\nFigure 9. Latent 369585 outperforms a dense SAE probe and lo-\ngistic regression when testing on mislabeled CoLA examples.\nsentences with target = 1 if the sentence is grammatically\ncorrect and 0 otherwise. Latent 369585 from the width =\n1M SAE has a test AUC of 0.76 and appears to fire on\nungrammatical text. Surprisingly, when we look at prompts\nthat latent 369585 fires on, those that it “disagrees” with the\nlabels on (ones that are labeled grammatical), often appear\nto be ungrammatical. Although we first found this result\nwith SAE probes, we later found that logistic regression was\nalso capable of the same identification. In Table 7, we show\nthe sentences that a baseline logistic regression classifier\nand latent 369585 mark as ungrammatical while the label is\ngrammatical; most such sentences are truly ungrammatical.\nThus, both SAE and baseline methods lead us to hypothesize\nthat the CoLA dataset is partially mislabeled.\nTo test our hypothesis, we choose 3 LLMs - GPT-4o (Ope-\nnAI), Claude-3.5-Sonnet-New, and Llama-3.1-405B In-\nstruct (Grattafiori et al., 2024) - to judge the linguistic ac-\nceptability of 1000 random CoLA prompts. We take the\nLLM majority vote as the ensembled, clean label. This is\nanalogous to the experiment performed in Warstadt et al.\n(2019) that validated the CoLA dataset by ensembling the\npredictions of native English speakers, but with LLM judges\ninstead of English speakers. We find that ˜25% of CoLA la-\nbels are mislabeled by this metric (higher than the 13%\ndisagreement rate from Warstadt et al. (2019)). Notably, the\nTable 7 sentences are identified as ungrammatical by the\nensemble while being labeled as grammatical in CoLA.\nNext, we train a baseline logistic regression, a k = 128\ndense SAE probe, and the latent 369585 classifier on the\noriginal CoLA labels. We then test on a held-out set with\nthe clean, ensembled predictions. In Figure 9, we find that\nthe original labels, the baseline classifier, the dense SAE\nprobe, and latent 369585 all have about the same accuracy\non the clean labels. Remarkably, when we test all classifiers\non examples which were misclassified in CoLA (where the\nensembled labels disagree with the original labels), we find\nthat the latent 369585 classifier outperforms the dense SAE\nprobe, which itself outperforms the baseline considerably.\n4.3.2. AI VS HUMAN\nWe also investigate 110 aimade humangpt3, which tests\nif probes can distinguish between human and ChatGPT-\n7\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse Probing105150:\nperiods\n116246:\nspaces\n3764:\nspaces\nLatent\n0\n10\n20\n|Mean Act Difference|\n(AI - Human)\nperiod quest.\nmark\nspace excl.\npoint\nLast Token\n0\n500\n1000\n1500\nTotal Activation Human\nAI\nFigure 10. Left: Top 3 latents (w/ descriptions) by mean activation\ndifference between AI generated and human SAE latent activations.\nRight: Histograms of the identity of the 4 most common last\ntokens in AI generated and human text.\n3.5 generated text. Latent 105150 has an AUC of 0.82\non this task, yet appears to fire primarily on periods and\npunctuation.2 Since this latent is syntactical and unrelated\nto the content of the dataset, we hypothesize it represents a\nspurious correlation in the dataset.\nWe examine the final token of each prompt in the dataset\nand indeed find a spurious feature: AI text is more likely to\nend in a period while human text is more likely to end in a\nspace (see Figure 10). Although we discovered this by inves-\ntigating SAE latents, we could reach the same conclusion\nwith baselines: in Table 8 (Appendix G.4), we apply the\nlogistic regression classifier to 2.8 million tokens from the\nPile, and find that it is most active on punctuation tokens.\n5. Why Didn’t This Work: Illusions of SAE\nProbes\nBricken et al. (2024a) report that SAE probes outperform\nbaselines slightly. A natural question is why our experi-\nments fail to replicate this result. One possible explanation\nis that Bricken et al. (2024a)’s findings were based on a\nsingle dataset, whereas we evaluate across multiple datasets.\nWe find that SAE probes outperform baselines on only a\nsmall subset of these datasets (2.2%, see Figure 11). It is\npossible that the dataset used by Bricken et al. (2024a) falls\nwithin this subset; however, without access to it, we cannot\nconfirm this.\nA separate explanation involves a potential illusion due to\ninsufficiently strong baselines. Unlike our work, Bricken\net al. (2024a) uses multi-token SAE probing: they aggregate\nthe maximum value for each latent across all prompt tokens,\nwhile we only use last token latents. Crucially, they simi-\nlarly max-pool each model dimension across prompt tokens\nfor their comparative baseline, even though activations are\nunlikely to have privileged dimensions (in Table 9 we show\nthat pooling activations in this way works poorly). We im-\nplement multi-token SAE probing using max-aggregation\n2See Neuronpedia for examples.0\n20\n40\n60\nWin Rate Percentage\n> 0.005 AUC 56.5%\n2.2%\nFAIR\nBoth = Last\n39.1%\n19.6%\nILLUSION\nBaseline = Last\nSAE = Pool\n34.8%\n8.7%\nFAIR\nBoth =\nQuiver(Pool, Last)\nBaseline > SAE SAE > Baseline\nFigure 11. Illustration of a potentially misleading result we find:\nwhen comparing activation probes to pooled SAE probes, SAE\nprobe win rate increases considerably, but adding in a “pooled”\nattention-inspired probe brings down the SAE win-rate.\non 60 random datasets from our list and k = 128; see Ta-\nble 9 for full results on these datasets. We also implement a\nbetter baseline: we train attention-pooled probes of the form\nsoftmaxt∈[1,CtxLen]{Xl\nt · q} · Xl · v (2)\nfor q, v ∈ Rd. We find that when compared to the last-token\nbaseline, max-pooled SAE probes win 19.6% of the time, a\nconsiderable improvement over win rate of last-token SAE\nprobes (2.2%, see Figure 11). However, when implementing\nattention-pooled baselines and using the quiver of arrows\napproach to select between pooled and last-token strategies\nfor SAE probes and baselines, the SAE probe win rate drops\nmore than 50% to 8.7% (see Figure 11).\n6. Assessing Improvements in SAE\nArchitectures\nWhile SAE probes do not robustly outperform baselines,\na separate interesting question is whether recent SAE ar-\nchitectural developments have improved SAE probing per-\nformance. We investigate this question by examining the\nperformance of eight different SAE architectures released\nin the last two years, as outlined in Table 2.\nWe use width = 16k Gemma-2-2B layer 12 SAEs with a\nvariety of L0 values trained by Karvonen et al. (2024b). We\ncreate k = 16, 128 SAE probes on all regimes and datasets\nfor each architecture, and additionally k = 1 for standard\nconditions. In Figure 12, we plot each SAE architecture’s\nk = 16 probe performance in standard conditions. We\nfind the average test AUC of each SAE and then take the\nmax across L0 for each SAE architecture (using a single\nL0 value is somewhat noisy, see Appendix I). This metric\ntests how expressive individual SAE latents are for different\nSAE architectures. While we see a slight positive trend\nfor probing performance with more recently released SAE\narchitectures, the effect is not statistically significant. For\nplots in all regimes, see Appendix I; in other regimes there\n8\n\nAre Sparse Autoencoders Useful? A Case Study in Sparse Probing\nTable 2. Timeline of SAE Architecture Improvements.\nSAE Architecture Publication Date\nReLU (original)\nBricken et al. (2023)","truncated":true,"next_offset":36000,"note":"More text follows; call again with offset 36000 to continue."}