Step 8: tool results

read_arxiv_source

{
  "file": "final_version.tex",
  "offset": 20000,
  "arxiv_id": "2505.20063",
  "max_chars": 10000
}

{"ok":true,"arxiv_id":"2505.20063","file":"final_version.tex","kind":"text","text":"sting they are predominantly input-aligned. Later layers ($66$--$100$\\% of model depth) show an opposite trend: input scores drop to near-zero, while output scores increase significantly. These later-layer features no longer reflect the tokens they are activated on, but instead align with the tokens they promote in the model’s output, indicating a shift toward output-aligned behavior. \nInterestingly, middle layers exhibit low scores for both metrics, suggesting that these features may play intermediate roles that are neither purely input-aligned nor strongly output-promoting. \n\n\nFor Llama-3.1 we do not observe any trend in early layers of the model (\\Cref{app:llama_pythia}), likely due to limitations of applying the logit lens to early layer representations \\cite{nostalgebraist}. \nAt around $50$\\% of the model's depth we begin to observe non-zero values for both scores, with low values of input scores and increasingly growing values of output scores as layers progress, as in the Gemma-2 results. \n\n\nFor Pythia, we observe a slightly different trend (\\Cref{app:llama_pythia}). \nWhile output score gradually increases around $50$\\% of the model's depth as expected, the input score is mainly zero for most of the tested features. \nThis may be due to the fact that this model is significantly smaller compared to the other models we examined ($70$ million parameters compared to $2$--$9$ billion), which may lead it to parse and encode information differently within its latent space. \n\n\nInterestingly, and unlike Llama-3.1, early layers in both Gemma models and Pythia seem to be interpretable with the logit lens. \nWe find that many features promote a coherent and human-understandable set of tokens, as reflected by high input scores as early as layers 0 and 1. (See \\Cref{app:input_features} for examples.)\n\n\n\n\n\\section{Identifying Features for Steering}\n\\label{sec:steering}\n\nThe output score measures the alignment between the effects of SAE features on the model's output distribution and the expected set of tokens---in our case, their top logit lens tokens. \nIn this section we hypothesize that features with high output scores are more effective for steering.\nTo test this, we evaluate generation success when filtering out features with low output scores at different thresholds. \n\n\\begin{figure}[t]\n \\centering\n    \\begin{subfigure}{0.49\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/os_th_2b.png}\n        \\caption{Gemma-2-2B.} \n    \\end{subfigure}\n    \\begin{subfigure}{0.49\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/os_th_9b.png}\n        \\caption{Gemma-2-9B.} \n    \\end{subfigure}\n\n \\caption{\\textcolor{magenta}{Magenta} indicates the mean generation success@20 when filtering out features with output scores below different thresholds.\n \\textcolor[HTML]{08A290}{Green} indicates the mean generation success@20 after filtering randomly sampled sets of features of the same size.\n Filtering results in significant increase in generation success.}\n \\label{fig:os_th}\n\\end{figure}\n\n\n\\subsection{Experimental Setup}\n\\label{subsec:steering_setup}\nWe use 50 prompt prefixes and generate up to 20 tokens, obtaining 50 generated texts for each feature (more details in \\Cref{app:steering}).\nFor each feature we calculate the mean generation success across the generated texts, and filter out steering factors leading to generation success greater than $3$. \nIntuitively, the generation success measures the rate in which the model generates concept-related tokens. \nBased on early experiments, we find that 3 is a good upper value that balances steering and coherence.\nWe choose the optimal steering factor as the one that maximizes $\\frac{\\text{Gen Success@20}}{\\text{Perplexity}}$, where we normalize both metrics to a $0$--$1$ range by dividing with the maximum value across all data samples. \n\n\\subsection{Qualitative Results}\n\\Cref{fig:steering} demonstrates steering with two features from Gemma-2-2B: one having a high input score and low output score (an input feature), and the other, an output feature, having a high output score and low input score.\nSteering using the input feature fails to meaningfully influence the generation. When using a high steering factor, this results in repeatedly generating tokens related to the feature’s activation, as if continuing from the word ''primary''.\nIn contrast, steering using a feature with high output score and low input score using an optimal steering factor yields fluent and semantically coherent completions. Additional examples are shown in \\Cref{app:steering}.\n\n\\begin{figure}[t]\n \\centering\n     \\begin{subfigure}{0.49\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/late_os_th_2b.png}\n        \\caption{Gemma-2-2B.} \n    \\end{subfigure}\n    \\begin{subfigure}{0.49\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/late_os_th_9b.png}\n        \\caption{Gemma-2-9B.} \n    \\end{subfigure}\n\n \\caption{Even in later layers of the model (16--25 for Gemma-2-2B and 24--41 for Gemma-2-9B), filtering features with low output scores increases mean generation success.\n\\textcolor{magenta}{Magenta}: filtering by output scores. \\textcolor[HTML]{08A290}{Green}: filtering random sets of features of the same size.}\n \\label{fig:th_late_layers}\n\\end{figure}\n\n\\subsection{Quantitative Results}\n\\Cref{fig:os_th} shows the mean generation success @ 20 of steered generations when filtering out features with output scores below varying thresholds (\\textcolor{magenta}{magenta}) on Gemma-2-2B and Gemma-2-9B. \nAs the threshold increases, performance improves steadily, indicating that features with higher output scores consistently lead to more successful steering. \nWe observe an increase in the mean generation success score from around $0.5-0.6$ for both models without any filtering, to $1.1$--$1.4$ using a threshold of $0.9$. A threshold of $0.01$ is sufficient for filtering out about $60\\%$ of the features, increasing the mean generation success by around $0.4$ points. \nWe compare this against a random baseline (\\textcolor[HTML]{08A290}{green}): filtering randomly sampled subsets of features of the same size does not lead to any significant improvements (results are average of 10 random samples per subset size). \nLlama-3.1 and Pythia show similar trends; see \\Cref{app:llama_pythia}.\n\n\nThe results in \\Cref{sec:roles} suggest that features with high output scores occur predominantly in later layers. \n\\Cref{fig:th_late_layers} shows the generation success when filtering based on the output score, evaluated only on features from later layers of the model: 16--25 for Gemma-2-2B and 24--41 for Gemma-2-9B.\nTaking only features from later layers, \\mbox{Gemma-2-2B} and Gemma-2-9B achieve generation success scores of about $0.8$.\nBy considering only top-scoring features, the mean generation success increases to around $1.1$--$1.4$ for both models. \nThese results show that even within these later layers, the output score is a useful tool for filtering out features that lead to poor steering results. \n\n\\begin{table}[]\n\\small\n\\begin{tabular}{llcc}\n \\toprule\n  &  & \\multicolumn{2}{c}{Gemma-2-9B-it} \\\\\n  \\cmidrule(lr){3-4}\n  &  & \\multicolumn{1}{c}{L20} & \\multicolumn{1}{c}{L31} \\\\\n\\midrule\nOur results &\n$\\text{SAE}_{(S_{out} \\geq 0.1)}$ & 0.546 & \\underline{0.470} \\\\\n  & $\\text{SAE}_{(S_{out} \\geq 0.01)}$  & 0.338 & 0.454 \\\\\n  & $\\text{SAE}_{(S_{out} \\geq 0.001)}$ & 0.373 & 0.415 \\\\\n  & $\\text{SAE}_{(S_{out} \\geq 0.0001)}$  & 0.325 & 0.401 \\\\\n  & $\\text{SAE}_{(\\text{No Filter})}$  & 0.293 & 0.387 \\\\\n\\midrule\n\\multirow{3}{*}{\\begin{tabular}[c]{@{}l@{}}AxBench \\\\\nreported \\\\\nresults \\\\ (Wu et al. 2025)\\end{tabular}} & \\cellcolor[HTML]{f0f0f0} Prompt & \\cellcolor[HTML]{f0f0f0} \\textbf{1.075}  & \\cellcolor[HTML]{f0f0f0} \\textbf{1.072}  \\\\\n  & \\cellcolor[HTML]{f0f0f0} LoReFT & \\cellcolor[HTML]{f0f0f0} 0.777 & \\cellcolor[HTML]{f0f0f0} 0.764 \\\\\n  & \\cellcolor[HTML]{f0f0f0} LoRA & \\cellcolor[HTML]{f0f0f0} 0.602 & \\cellcolor[HTML]{f0f0f0} 0.580 \\\\\n  & ReFT-r1 & \\underline{0.630} &  0.401 \\\\\n  &  DiffMean &  0.322 &  0.158 \\\\\n  &  SAE  &  0.191 &  0.140 \\\\\n  &  SAE-A &  0.186 &  0.143 \\\\\n\\bottomrule\n\\end{tabular}\n\\caption{\\textbf{Results on the Concept500 dataset from AxBench on instruction-tuned Gemma-2-9B.} (Top) Results when steering with SAEs after filtering out features with $S_{out}$ lower than different thresholds. (Bottom) Results reported by \\citet{wu2025axbench}. \\textbf{Bold} indicates the best score, \\underline{underline} indicates the best score among representation-based methods. \\colorbox[HTML]{f0f0f0}{Grey} indicates non-representation-based methods.\n% Note that in our experiments, SAEs without any filtering achieve a higher score than reported in AxBench; see Footnote~\\ref{footnote:axbench}.\nAfter filtering based on output scores, SAEs achieve the best score among representation-based methods at L31, and reach 90.7\\% of the best method's performance at L20.}\n\\label{table:axbench}\n\\end{table}\n\n\n\\subsection{Evaluation on AxBench}\nAxBench was recently proposed as a dataset to evaluate steering methods \\cite{wu2025axbench}. \nThey compare steering with SAE features to other methods (including supervised methods), and find SAE features relatively ineffective.\nHowever, we believe this is partially due to a non-principled selection of SAE features; we propose to remedy this using the output score.\n\nWe evaluate our findings on instruction-tuned Gemma-2-9B using the Concept500 dataset of \\citet{wu2025axbench}, which includes 1000 $\\langle$concept, SAE feature$\\rangle$ pairs from layers 20 and 31 of the model. \nAs in AxBench, we randomly sample 10 instructions for each concept-feature pair from instruction datasets aligned with the concept’s genre.\nFive are used to select the optimal steering factor (as detailed in \\Cref{subsec:steering_setup}), and the remaining five are used exclusively for evaluation.\nWe evaluate the steered te","chars":10000,"total_chars":64852,"offset":20000,"truncated":true,"next_offset":30000}

read_arxiv_source

{
  "file": "final_version.tex",
  "offset": 50000,
  "arxiv_id": "2505.20063",
  "max_chars": 15000
}

{"ok":true,"arxiv_id":"2505.20063","file":"final_version.tex","kind":"text","text":"sholds (\\textcolor{magenta}{magenta}) on Llama-3.1 and Pythia. \nSimilarly to the Gemma models, as the threshold increases, performance improves steadily, indicating that features with higher output scores consistently lead to more successful steering. \n\n\\begin{figure*}\n    \\centering\n    \\begin{subfigure}{0.4\\linewidth}\n        \\centering\n        \\includegraphics[width=0.9\\linewidth]{figures/llama31_llama31_roles.png}\n        \\caption{Llama-3.1-8B.} \n    \\end{subfigure}\n    \\begin{subfigure}{0.4\\linewidth}\n        \\centering\n        \\includegraphics[width=0.9\\linewidth]{figures/pythia70_pythia70_roles.png}\n        \\caption{Pythia-70m.} \n    \\end{subfigure}\n    \n    \\caption{\n        \\textbf{Input and output scores across layers in Llama-3.1-8B and Pythia-70m.}\n        The solid lines represent the median \\textcolor{blue}{input} score (blue) and \\textcolor{magenta}{output} score (magenta), while the shaded regions denote the interquartile range (25th to 75th percentile), capturing the variability across features within each layer.\n        In these models we observe that high output scores emerge in later layers, while input score is mostly zero across all layers.\n    }\n    \\label{fig:app_roles}\n\\end{figure*}\n\n\n\\begin{figure*}\n \\centering\n    \\begin{subfigure}{0.4\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/llama31_llama31_output_score_rand_th.png}\n        \\caption{Llama-3.1-8B.} \n    \\end{subfigure}\n    \\begin{subfigure}{0.4\\linewidth}\n        \\centering\n        \\includegraphics[width=\\linewidth]{figures/pythia70_pythia70_output_score_rand_th.png}\n        \\caption{Pythia-70m.} \n    \\end{subfigure}\n\n \\caption{\\textcolor{magenta}{Magenta} indicates the mean generation success@20 when filtering out features with output scores below different thresholds.\n \\textcolor[HTML]{08A290}{Green} indicates the mean generation success@20 after filtering randomly sampled sets of features of the same size.\n Filtering results in significant increase in generation success.}\n \\label{fig:app_os_th}\n\\end{figure*}\n\n\n\n\n\n\\section{Identifying Input Features}\n\\label{app:input_features}\n\\Cref{app_table:input_features} shows examples of features from layers 0 and 1 having high input scores. For each feature, the table includes it's top-5 logit lens tokens as well as examples for input texts that activated this feature. The tokens where the feature activated most strongly are marked using an \\underline{underline}.\n\n\n \\begin{table*}[]\n \\small\n \\label{app_table:input_features}\n\\begin{tabular}{lllll}\n\\toprule\nLayer & Feature & Input Score & Top-5 Logit Lens Tokens & Activated Text \\\\\n\\midrule\n0 & 725 & 0.822 & \n\\begin{tabular}[c]{@{}l@{}}\n{[}'\\_she', '\\_It', \\\\\n'\\_we', 'she', '\\_OHA'{]}\n\\end{tabular} & \n\\begin{tabular}[c]{@{}l@{}}\n... euphoria and uncertainty.\\underline{ He} asked himself, “ ...\\\\\n...\\underline{ She} also recalls the expeditions ...\\\\\n... with a win.\\underline{ He} said he's ...\n\\end{tabular} \\\\ \\\\\n0 & 14772 & 0.889 & \n\\begin{tabular}[c]{@{}l@{}}\n{[}'inning', 'inb', 'inl',\\\\\n'IN', 'inare'{]} \n\\end{tabular} & \n\\begin{tabular}[c]{@{}l@{}}\n... deposition of both elast\\underline{in} and fib rill\\underline{in} ...\\\\\n... I. Man\\underline{in}, Three-dimensional ...\\\\\n... $, $k\\textbackslash{}\\underline{in} \\{1,\\textbackslash{}dots ...\n\\end{tabular} \\\\ \\\\\n1 & 4413 & 0.844 & \n\\begin{tabular}[c]{@{}l@{}}\n{[}'cleared', 'cle', ' clearing', \\\\\n' clearance', ' cleaned'{]}\n\\end{tabular} & \n\\begin{tabular}[c]{@{}l@{}}\n... set. Once we\\underline{ cleared} all the debris, ...\\\\\n...\\underline{ clearing} room for a future ...\\\\\n... If this were only\\underline{ cleared} away,\" They said ...\n\\end{tabular}\\\\ \\\\\n1 & 3110 & 0.867 & \n\\begin{tabular}[c]{@{}l@{}}\n{[}'\\_deltas', 'Delta', \\\\\n'DeltaTime', \\textbackslash{}Delta, 'eltas'{]}\n\\end{tabular} & \n\\begin{tabular}[c]{@{}l@{}}\n... large potential difference (\\$\\underline{\\textbackslash{}Delta}\\$ V = ...\\\\\n... atmospheric thickness, \\$\\underline{\\textbackslash{}Delta} z\\_{eff}\\$, ...\\\\\n... finite temperature bias \\$\\underline{\\textbackslash{}Delta} T\\$ generates a ...\n\\end{tabular} \\\\\n\\bottomrule\n\\end{tabular}\n \\caption{Examples of features with high input scores in early layers. Activated token are marked with an \\underline{underline}.}\n\\end{table*}\n\n\\section{Neutral Prompt Selection}\n\\label{app:neutral_prompt}\nOur output score relies on the use of a single neutral prompt as a prefix to the model's generation. \nTo evaluate the robustness of our score to the specific choice or neutral prompt we randomly selected 10 features from each layer of Gemma-2-2B (240 features overall) and computed their output scores using all $50$ of the neutral prompts in \\Cref{app_table:prefixes}.\n We find that the correlation between the output score computed with the original prompt and the mean score is $0.9557$, which is extremely high and indicates that the exact choice of the prompt has almost no impact on the results. \n\n\n\\section{Steering Details}\n\\label{app:steering}\nFor our main experiments, we test steering factor values $s \\in \\{ 0.2, 0.4, 0.8, 1.2, 1.6, 2.0, 3.0, 4.0, \\\\ 6.0, 8.0, 10.0, 20.0\\} $.\nWe generate 20 tokens using a temperature of $0.7$ after each of the prefixes listed in \\Cref{app_table:prefixes}.\n\n\\begin{table*}[]\n\\centering\n\\resizebox{0.95\\linewidth}{!}{%\n\\begin{tabular}{l|l|l}\n\\toprule\n''Findings show that'' & ''It’s no surprise that'' & ''It's been a long time since'' \\\\\n''I once heard that'' & ''Have you ever noticed that'' & ''In my experience,'' \\\\\n''Then the man said:'' & ''I couldn't believe when'' & ''The craziest part was when'' \\\\\n''I believe that'' & ''The first thing I heard was'' & ''If you think about it,'' \\\\\n''The news mentioned'' & ''Let me tell you a story about'' & ''I was shocked to learn that'' \\\\\n''She saw a'' & ''Someone once told me that'' & ''For some reason,'' \\\\\n''It is observed that'' & ''It might sound strange, but'' & ''I can’t help but wonder if'' \\\\\n''Studies indicate that'' & ''They always warned me that'' & ''It makes sense that'' \\\\\n''According to reports,'' & ''Nobody expected that'' & ''At first, I didn't believe that'' \\\\\n''Research suggests that'' & ''Funny thing is,'' & ''That reminds me of the time when'' \\\\\n''It has been noted that'' & ''I never thought I'd say this, but'' & ''It all comes down to'' \\\\\n''I remember when'' & ''What surprised me most was'' & ''One time, I saw that'' \\\\\n''It all started when'' & ''The other day, I overheard that'' & ''I was just thinking about how'' \\\\\n''The legend goes that'' & ''Back in the day,'' & ''Imagine a world where'' \\\\\n''If I recall correctly,'' & ''You won’t believe what happened when'' & ''They never expected that'' \\\\\n''People often say that'' & ''A friend of mine once said,'' & ''I always knew that'' \\\\\n''Once upon a time,'' & ''I just found out that'' & \\\\\n\\bottomrule\n\\end{tabular}\n}\n\\caption{Neutral prefixes used for generation for main steering experiments.}\n\\label{app_table:prefixes}\n\\end{table*}\n\n\n\\section{AxBench Details}\n\\label{app:axbench}\n\\paragraph{Model Instructions.} Each concept in the Concept500 dataset is annotated as either ''text'', ''math'', or ''code''.\nFollowing their setup we randomly sample 10 instructions per concept from instruction datasets that match the concept genre: Free Dolly dataset for text instructions \\cite{DatabricksBlog2023DollyV2}, GSM8K for math \\cite{cobbe2021gsm8k}, and Alpaca-Eval for code \\cite{alpaca_eval}. \n\n\\paragraph{Steering Details.} We generate up to 128 tokens per instruction, with a temperature of $0.7$, using steering factor values of $\\{0.4, 0.8, 1.2, 1.6, 2.0, 3.0, 4.0, 6.0, 8.0, 10.0, 20.0, \\\\40.0, 60.0, 100.0\\}$. \nFor each concept, five instructions are used to choose the optimal steering factor (as described in \\ref{subsec:steering_setup}), and the rest are used for evaluation.\n\n\\paragraph{Metrics.} We evaluate the steered texts using the metrics defined by \\citeauthor{wu2025axbench}: (1) the concept score ($cs$) measures if the concept was incorporated in the generated text, (2) the fluency score ($fs$) measures the coherency of the text, and (3) the instruct score ($is$) measures the alignment of the generated text with the given instructions. \nFor each metric ($m \\in \\{\\text{cs}, \\text{fs}, \\text{is}\\}$) and steered text $s$, an external rater returns a discrete score of either 0, 1, or 2: $m(s) \\in \\{0, 1, 2\\}$. \nAs the external LLM rater, we use Claude 3.7 sonnet (2025-02-19) \\cite{claude2024}.\nThe prompts for all metrics are given in Tables~\\ref{app_table:concept_score}, \\ref{app_table:instruct_score}, and\\ref{app_table:fluency_score}.\n\n\nFor each concept $c$, we compute each metric over the five test instructions and take the mean: $m(c) = \\frac{\\sum_{s \\in S}{m(s)}}{|S|}$. The overall score of a concept is the harmonic mean of the three scores: $(\\text{cs}(c), \\text{fs}(c), \\text{is}(c))$. \n\n\nThe cost of obtaining this score for the tested features was 65 USD.  \n\n\n\\paragraph{Baseline Methods.} \n\\Cref{table:axbench} shows the reported results of steering using various methods, as achieved by \\citet{wu2025axbench}. We provide additional details on each of these methods:\n\\begin{itemize}\n    \\item \\textbf{Prompt.} Given a concept, an external LLM generates a steering prompt. This prompt is then pre-pended to the actual instructions.\n    \\item \\textbf{LoRA.}\n    A parameter-efficient finetuning method \\cite{xu2024lora}, trained to minimize the language modeling loss on a set of positive examples that contain the concept.\n    \\item \\textbf{LoReFT.} Low-rank representation finetuning \\cite{wu2024reft}, with the same objective as LoRA.\n    \\item \\textbf{ReFT-r1.} A weakly supervised approach based on \\citet{wu2024reft} \\cite{wu2025axbench}. ReFT-r1 jointly learns concept detection and steering on supervised data by combining the training objectives of linear probing and supervised steering.\n    \\item \\textbf{Difference-in-means.} Difference-in-means (DiffMean) \\cite{pmlr-v48-larsen16} uses the difference between averaged representations from two classes of inputs as a steering vector.\n    \\item \\textbf{SAE-A.} SAEs with AUROC selection \\cite{wu2025axbench}.  For each feature, its max-pooled activations are computed over a set of positive training examples. Top-scoring features, based on the AUROC over the dataset given true labels, are used for steering.\n\n\\end{itemize}\n\n\\paragraph{License.} AxBench and Alpaca-Eval are published under an Apache license. GSM8K is published under MIT license. Dolly is published under Creative Commons Attribution Share Alike 3.0 license. All artifacts are used in compliance with their license.\n\n\n\\begin{table*}[]\n\\begin{tabular}{l}\n\\toprule\n\\textbf{Concept Score} \\\\\n\\midrule\n\\begin{tabular}[c]{@{}l@{}}{[}System{]}\\\\\nPlease act as an impartial judge and evaluate whether the specified concept is presented in the \\\\\nsentence fragment provided below. Focus solely on whether the concept is clearly incorporated, \\\\\nwithout regard for grammar or logical coherence. Begin your evaluation by providing a short \\\\\nexplanation of whether the concept is clearly incorporated. Point out references to the concept \\\\\nin the fragment, noting any minimal or awkward incorporations. \\\\\nAfter providing your explanation, rate the concept’s relevance on a scale \\\\\nfrom 0 to 2, where 0 indicates the concept is not present at all,\n1 indicates the concept is somewhat \\\\\npresent but minimally or awkwardly incorporated, and 2 indicates the concept is more fully \\\\ \nand effectively incorporated, with stronger and more natural integration. \\\\\nProvide your rating using this exact format: “Rating: {[}{[}score{]}{]}”.\\\\ \\\\\n{[}Concept Start{]}\\\\\n{[}Concept goes here{]}\\\\\n{[}Concept End{]}\\\\\n{[}Sentence Fragment Start{]}\\\\\n{[}Sentence goes here{]}\\\\\n{[}Sentence Fragment End{]}\\\\\n\\bottomrule\n\\end{tabular}\n\\end{tabular}\n\\caption{Prompt used to compute the concept score.}\n\\label{app_table:concept_score}\n\\end{table*}\n\n\n\\begin{table*}[]\n\\centering\n\\begin{tabular}{l}\n\\toprule\n\\textbf{Instruct Score} \\\\\n\\midrule\n\\begin{tabular}[c]{@{}l@{}}{[}System{]}\\\\\nPlease act as an impartial judge and evaluate whether the sentence fragment provided below \\\\\nis related to the instruction. Focus solely on the degree of relatedness in terms of \\\\\ntopic, regardless of grammar, coherence, or informativeness.\nBegin your evaluation by \\\\\nproviding a brief explanation of whether the sentence is related to the instruction, and \\\\\npoint out references related to the instruction. After providing your explanation, rate \\\\\nthe instruction relevance on a scale from 0 to 2, where 0 indicates the sentence is unrelated \\\\\nto the instruction, 1 indicates it is somewhat related but only minimally or indirectly \\\\\nrelevant in terms of topic, and 2 indicates it is more clearly and directly \\\\\nrelated to the instruction. \\\\\nProvide your rating using this exact format: “Rating: {[}{[}score{]}{]}”.\n\\\\ \\\\\n{[}Instruction Start{]}\\\\\n{[}Instruction goes here{]}\\\\\n{[}Instruction End{]}\\\\\n{[}Sentence Fragment Start{]}\\\\\n{[}Sentence goes here{]}\\\\\n{[}Sentence Fragment End{]} \\\\\n\\end{tabular}\n\\\\\n\\bottomrule\n\\end{tabular}\n\\caption{Prompt used to compute the instruct score.}\n\\label{app_table:instruct_score}\n\\end{table*}\n\n\n\\begin{table*}[]\n\\begin{tabular}{l}\n\\toprule\n\\textbf{Fluency Score} \\\\\n\\midrule\n\\begin{tabular}[c]{@{}l@{}}{[}System{]}\\\\\nPlease act as an impartial judge and evaluate the fluency of the sentence fragment provided \\\\\nbelow. Focus solely on fluency, disregarding its completeness, relevance, coherence with any \\\\\nbroader context, or informativeness. Begin your evaluation by briefly describing the fluency \\\\\nof the sentence, noting any unnatural phrasing, awkward transitions, grammatical errors, or \\\\\nrepetitive structures that may hinder readability. After providing your explanation, rate the \\\\\nsentence’s fluency on a scale from 0 to 2, where 0 indicates the sentence is not fluent and \\\\\nhighly unnatural (e.g., incomprehensible or repetitive), 1 indicates it is somewhat fluent but \\\\\ncontains noticeable errors or awkward phrasing, \\\\\nand 2 indicates the sentence is fluent and almost perfect. \\\\\nProvide your rating using this exact format: “Rating: {[}{[}score{]}{]}”. \\\\ \\\\\n\n{[}Sentence Fragment Start{]}\\\\\n{[}Sentence goes here{]}\\\\\n{[}Sentence Fragment End{]}\n\\end{tabular} \\\\\n\\bottomrule\n\\end{tabular}\n\\caption{Prompt used to compute the fluency score.}\n\\label{app_table:fluency_score} \n\\end{table*}\n\n\n\\section{Computational Requirements}\nThe experiments in this work were conducted using an NVIDIA A$40$ node with $8$ $48$GB GPUs.\nThe complete set of experiments, including initial exploration, consumed 300-500 GPU hours.\n\nComputing the output score for a single feature takes 6 seconds on average on a single A$40$ $48$GB GPU. Computing the input scores given pre-computed activations takes negligible time. \n\n\\end{document}\n\n\n\n","chars":14852,"total_chars":64852,"offset":50000,"truncated":false,"next_offset":null}