Methodological Disaster Zone

The paper Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines is a response to the AxBench paper, AxBench: Steering LLMs? Even simple baselines outperform sparse autoencoders. Aryaman Arora tweeted “has anyone ever written a diss track of your paper”. I responded: “Like any good advisor, I felt duty-bound to defend @aryaman2020 and @ZhengxuanZenWu in this rap battle. However, the SAE diss track I wrote was so devastating as to be unanswerable, so I decided to graciously balance things out with a second verse dissing causal interp”. The result was this song (created with Suno), posted on Twitter on June 9, 2026.

I tweeted “I wrote the lyrics as an homage to ‘Hypnotize’ by The Notorious B.I.G., but Biggie apparently cannot be imitated, so the @suno version uses a totally different style”. The structure of the song follows “Hypnotize” closely and borrows a few of its rhymes. Biggie is inimitable.

The conceit of this song is that there is a beef within the interp community between SAE advocates and causal interp advocates. This line is meant to evoke a famous beef in the history of hip-hop between Biggie and Tupac Shakur. Biggie was a Brooklyn native, and Tupac was associated with the Bay Area city of Oakland. Stanford is in the Bay Area, and so you might expect the song to be aligned with Tupac. Instead, this line seeks to convey a note of harmony before the disses begin.

To paraphrase Tony Soprano, “There is no interpretability Mafia”.

This line conveys my view that interp researchers do more hand-wringing about methods, beliefs, and minor research setbacks than any other field in AI. I am willing to say that this stems from the fact that the field is still figuring out what its true goals are, but I am not willing to concede that interp is in worse shape than other areas of AI. I would say that all areas would do well to be more introspective.

This is an exaggeration. The song disses only two classes of method within interp: sparse coding (represented by SAEs and, to a lesser extent, transcoders) and causal interventions (represented by DAS). It doesn’t touch attribution methods, supervised probing, or adversarial evaluations. All of them deserve at least their own verse, though.

At the time the song was written, post-training was one of the most influential and best-resourced areas of research within AI. At the same time, there were reports that supposed breakthroughs were the result of data contamination and, in any case, were not adding much to the base model, alongside papers reporting the opposite. Unlike with interp researchers, these events did not lead post-training researchers to declare that their field was in crisis. They simply carried on with their work and made progress.

In any case, as they say of The Control Verse, it’s an honor to be mentioned.

A direct reference to the first line of Biggie’s “Hypnotize”: “Hah, sicker than your average”. It also invokes the result that SAEs under-perform even simple baselines on AxBench, as well as the general criticism that advanced interp methods are often no better than simple data analysis and attribution methods.

This is the first line of the verse dissing SAEs, and the first diss is that training SAEs is more art than science.

The guiding intuition behind SAEs is fairly clear and precise. Assume neural networks store at least some of their features as potentially overlapping directions in their activation spaces. The job of the SAE is to disentangle these features so that they are represented by individual latent dimensions in the SAE’s representation space. SAEs do this in an unsupervised manner: they map the input activation vector to a large, sparse hidden representation h and then seek to reconstruct the input on the basis of h. The reconstruction is a weighted sum of the decoder’s columns, so each column acts as a candidate feature direction – a dictionary of features – and h determines which features are active and how strongly. The representation h is the focus for interpretability work, as it is meant to be the interpretable, disentangled view of the input x.

We seek SAE representations h in which only a small number of dimensions are active for any given input, because we think these will be more interpretable. Modern SAEs, in turn, incorporate a lot of sparsity-encouraging pressures: the activation function is a sparsity-inducing one like ReLU (as discussed later in this verse), input activation vectors are typically centered so that the network models only how they differ from the average, and the loss always includes at least one term designed to induce sparsity in h (e.g., the L1 (a.k.a. LASSO) penalty mentioned in the next line).

This all sounds reasonable and productive, but two conceptual issues immediately arise:

1. Faithfulness: In practice, the SAE is always lossy because of the push to make h sparse. If we drop this goal, then faithfulness is basically within reach; the dimensionality of h is always defined to be much larger than the dimensionality of x, and so the SAE could learn to approximate an identity map. However, this would not further any interpretability goal.

2. Uniqueness: Even with zero reconstruction error and a fixed sparsity level, the SAE’s features are not uniquely determined, which raises difficult conceptual questions of what the “true” features are. Unfortunately, the conditions under which a sparse dictionary-learning solution is fully identifiable are unlikely to arise in real-world contexts.

These issues reveal an inherent tension: sparsity trades off against faithfulness and exacerbates the uniqueness issue. In other words, the more sparsity you insist upon, the more interpretable the SAE features will seem, but the less trustworthy the SAE is likely to be. This is extremely delicate. As the line says, you end up having to proceed on instinct.

There are objective measures we can bring to bear on the faithfulness issue. The reconstruction error directly measures how different the input and output are. In addition, we could actually replace the input with its reconstruction in the original network and see how much this substitution changed the behavior of the network as a whole. However, perfect scores are unattainable, and they would not resolve the uniqueness issue anyway. It is, thus, perhaps no surprise that these metrics seem generally to take a backseat to how intuitively interpretable the SAE features are.

This is a bit of metonymy. LASSO is an L1-penalized regression method, and the L1 penalty is often used when training SAEs to encourage sparsity in the hidden representation h.

The mechanism here is to add a penalty proportional to the sum of the absolute values of h to the training loss. L1 will exacerbate the uniqueness issue discussed in connection with my previous annotation, because it simply pushes toward a small total magnitude for the values in h. This can be achieved in lots of ways. For example, a concept could be encoded in one dimension or split across several dimensions. If the total magnitude is the same in both cases, the L1 loss will be the same in both cases, even though the two scenarios surface very different features.

These lines touch on two activation functions that have been tried for SAEs in an effort to obtain faithful and interpretable latent representations (“your saviors”, but wryly): TopK and Gated SAEs.

The TopK paper refers to its core activation function as “k-sparse” after Makhzani and Frey 2013, which is a very early SAE proposal. This function keeps only the k largest activation values in the hidden representation of the SAE, zeroing out all the rest. This is noteworthy because it is more like a masking operation than a differentiable activation function: when doing weight updates during training, the gradients flow only through the k selected elements, with the gradients for the rest simply set to 0.

Quick aside: the SPINE paper (mentioned a bit later in this verse) builds on the sparse autoencoders of Ng 2011, a set of course notes (!) which might be the true OG SAE proposal. The SPINE group calls these “k-sparse autoencoders” (Ng does not use that term). However, Ng’s SAEs achieve something approximating k-sparsity via a term in the loss function, not with a top-k activation function, and SPINE modifies and extends this idea. In other words, SPINE’s (Ng’s) “k-sparse” is very different from Makhzani and Frey’s “k-sparse”.

Gated SAEs are even more fascinating from the perspective of language model research as a whole. The central idea is to replace the standard ReLU activation with one that separately learns which features are “on” and what magnitude those features should have. This is directly inspired by the proposals of Shazeer 2020 for using similar activation functions in the feed-forward layers of the Transformer. Shazeer’s proposals were widely adopted a few years later for LLM architectures in general, and this had the effect of turning the feed-forward layers of the Transformer into an SAE-like object, thereby reducing the need for a separate SAE at all. (This may be just a personal opinion – but see Arora et al. 2026.)

Nota bene: Describing these as “link functions” is confusing and should not be encouraged. This terminology comes from the study of generalized linear models in statistics, where it refers specifically to the function that relates the predictor to the expected value of the output. The accepted term in AI/ML is “activation function”. However, this passage is trying to keep pace with Biggie’s rate of internal rhymes, so it needs “link” and “shrink”. (I also like it that “shrink” can mean “psychiatrist” as well as “reduce in size/norm”.)

One of the primary selling points of SAEs has always been that they are unsupervised. This is meant to convey that they allow for free-form discovery of features. The presumption is that, in contrast, supervised methods require us to know ahead of time what we are looking for and are therefore less useful for exploration.

Inevitably, though, supervised signals creep in. This usually starts with researchers’ intuitions about what should happen. If those things don’t happen, they fiddle with the set-up and try again. In this way, they begin hill-climbing towards a particular outcome. This is an informal sort of supervised learning.

For SAEs, curated SAE training data has been another source of (what is in effect) supervision. This provides another avenue for steering the entire project toward specific desired outcomes. Makelov et al. 2024 quantify the potential gains here: SAEs trained on task-specific data outperform general-purpose SAEs on task-specific evaluations.

It is significant here that, at least in my view, the advertised line about free-form discovery is misleading. The hidden dimensionality of the SAE is fixed, whereas the space of all concepts is probably not even finite, and no one has established that the SAE will lawfully, or even reliably, find the most general, best represented, or most prevalent concepts in the target model. What about all the concepts that don’t emerge from the SAE training? Are they in the target model or not? The only way to address this with SAEs is to change some things about the SAE set-up and train again. This is very expensive and time-consuming.

LessWrong is an online forum devoted to discussions of rationalism across many domains. Why are SAEs associated with LessWrong? The web of interconnections is dense and complex, but I believe these are the highlights:

1. As the LessWrong about page says, “Historically, LessWrong was seeded by the writings of Eliezer Yudkowsky, an artificial intelligence researcher.” Yudkowsky is famous for his concerns that AI will lead to the end of the world in one way or another. Research in AI safety is intended to forestall these apocalyptic outcomes.
2. Interp and AI safety are intimately related. Tracing this association is difficult, but the “Best of LessWrong 2019” post by Evan Hubinger titled Chris Olah’s views on AGI safety is clearly a key link. It assembles arguments that interpretability enables auditing and deliberate model design, and that these improve our chances of catching misalignment. In essence, this seeks to justify investment in interp on the grounds that it will improve AI safety. In turn, LessWrong became a hub for interp research and discussions, and it subsequently became a home for a lot of influential SAE research.

Training “all night” is actually much too generous. SAEs are extraordinarily expensive as a technology. For example, it seems safe to say that reproducing the GemmaScope SAEs would cost over $200K in training compute alone (setting aside the costs of running the target model to create activations and storing those activations). In my view, this compromises the claim that they can be vehicles for free-form discovery. Along many dimensions, it is cheaper to train thousands or millions of tiny supervised models than it is to train one large SAE.

For the cost estimate: The GemmaScope paper says that the project “used over 20% of the training compute of GPT-3 (Brown et al., 2020), saved about 20 Pebibytes (PiB) of activations to disk, and produced hundreds of billions of sparse autoencoder parameters in total”. For just the training compute:

GPT-3 used g = 3.14 × 1023 FLOPs, so GemmaScope used s = 6.3 × 1022 FLOPs. Assume training on TPU v5e or comparable accelerators delivering t = 2 × 1014 peak FLOPs/sec. At typical ML utilization levels of 50% of peak, this is throughput of u = 1 × 1014. This gives s / u = 6.3 × 108 accelerator-seconds, and thus 175,000 accelerator-hours. At a blended cloud rate of $1.20–2.00/hour, compute for training is in the range $210,000–$350,000.

Note: my calculation assumes bf16, but I believe GemmaScope used 32-bit precision. This doesn’t affect the FLOPs, but rather only throughput – my throughput estimate may be too high, which would artificially reduce the cost estimate.

SPINE (SParse Interpretable Neural Embeddings) is the method developed by Subramanian et al. 2017, long before SAEs rose to prominence, and indeed before LLMs were really a thing. The inputs to their method are static (word2vec) vectors, but the goals are the same as for modern SAE efforts. They achieve sparsity via (1) two sparsity-inducing terms in their loss function and (2) a capped ReLU activation function (just like a standard ReLU but activation values larger than 1 are mapped to 1; often called “clipped ReLU” or “ReLU-1” these days).

SPINE is not the first proposal for SAEs. I cited Makhzani and Frey 2013 above, and SPINE itself extends and modifies the proposal for sparse autoencoders in Ng 2011. However, as far as I know, SPINE is the first application of SAEs to understanding learned vector representations of language, and thus I wanted to give them a shout-out. In general, all these early contributions deserve much more credit than they have gotten in this era when SAEs are (or were, until recently) cool.

For the period roughly May 23, 2024, to March 26, 2025, the field of interpretability was identified with SAEs. People would refer to “mech-interp” but have only SAEs in mind, which was a lot like talking with someone about carpentry only to discover, after a lot of apparent disagreements, that by “carpentry” they meant “hammers”. I often interacted with junior researchers who thought that SAEs were the only conceivable approach to interp and who were quite baffled when I said things analogous to “hammers might not be the best tool for cutting wood”.

Why did the field go all-in on SAEs? It was not because there was abundant experimental evidence in their favor. On August 5, 2024, I tweeted, “The rapid convergence around SAEs seems premature to me. There is currently no record of empirical successes to justify focusing on them to the exclusion of other options”. I believe this was, and remains, an uncontroversial statement. Can we reconcile all of this? I am skeptical.

As I noted above in connection with LessWrong, the background philosophy here is rationalism. I understand this to be a form of Bayesian rationalism that allows for strong priors (of mathematical elegance, etc.), as in classical rationalism, while also emphasizing the importance of updating those priors in a controlled way based on new evidence (classical empiricism).

If the mathematical case for SAEs were airtight, then this philosophical outlook might justify deep investment in them even before much experimental evidence had been amassed. However, as my comments on the first line of this verse make clear, the analytic argument for SAEs is shaky. Overall, then, I would contend that the Bayes-rational response would be a diversified portfolio across the field, not sudden field-wide convergence on one idea.

In this context, it was eye-opening to me that most negative evaluations of SAEs carried little weight. It was only when influential insiders to the SAE movement started to say negative things about SAEs that attitudes shifted. And, when they did shift, it was a shockingly abrupt – dare I say, irrational – shift, akin to concluding “carpentry is useless” when the evidence merely said “jigsaws are only useful for cutting irregular curves”.

This line requires an awkward synecdoche; I need “Claude” to stand in for “Anthropic”, which seemed to leave SAEs behind in March 2025 in favor of transcoders/crosscoders, another family of sparse interpretability methods.

This line belongs with the other activation functions at the start of the verse, but it was funny to include it here as though it were the culmination of some absurdity. JumpReLU is a threshold-based variant of ReLU: each latent dimension of the SAE hidden representation h has a threshold, and activation values below that value are zeroed out, with the rest passing through unchanged. This can be seen as capitalizing on the same insights that motivate Gated SAEs, discussed above in connection with the “gaters” lyric.

This line shouts out older dictionary learning methods that seem like conceptual cornerstones of the entire SAE effort, even if their specific details are not very central to present-day discussions.

K-SVD is close to modern SAEs conceptually, especially variants that impose a hard budget on the number of active dimensions in h (e.g., TopK discussed above), with the main difference being that K-SVD uses an iterative, alternating-minimization algorithm to find its dictionary, whereas SAEs use gradient descent.

It’s more of a stretch to group ICA with SAEs, but I liked how it sounded. Classical ICA is used only in situations where the input and dictionary have the same size, whereas the defining feature of SAEs is that they are overcomplete – more dictionary dimensions than input dimensions. (Overcomplete ICA variants come much closer to SAEs.) However, the two are united in aiming to decompose the input into a set of more meaningful (interpretable) latent components.

This line refers to a mission statement that Chris Olah and Adam Jermyn wrote for interp: Reflections on qualitative research (see also Interpretability as a natural science). Their argument is that, since the field is still trying to figure out what its goals are, it should be allowed space for free exploration, unencumbered by benchmarking, fixed metrics, or entrenched conventions for what counts as a result.

I am sympathetic with the overall perspective, but I am also wary of it. I started my career doing pure linguistics. That field still operates in this preparadigmatic mode. One of the most influential books in the history of the field is Noam Chomsky’s 1995 The Minimalist Program. Chomsky has consistently framed Minimalism as a “program”, not a framework or theory. It’s a running joke in the field that, more than 30 years later, it still hasn’t graduated from “program”, despite being the object of intense study by linguists ever since its publication.

The deeper problem with operating preparadigmatically is that it actually makes it harder for new ideas to break through. If you have benchmarks, and some iconoclastic idea does well on those benchmarks, it is likely to get a hearing. Without this mechanism, the field is subject to fashions and trends, as well as a rich-get-richer dynamic in which popular groups/ideas get attention no matter what they do while everyone else has to scramble to justify all their assumptions all the time.

The reverse dynamic – the one that Olah and Jermyn are worried about – is also very real: benchmarks can lead people to iterate on past ideas rather than boldly trying new things. I am worried about this as well, but I would offer the following pragmatic remark: we created AxBench on a shoestring budget in just a few months. Anthropic could create a new AxBench variant every single week, in response to new ideas and problems, without it even slowing them down or affecting their finances. In other words, we can operate with preparadigmatic freedom while still creating a forum for objective apples-to-apples comparisons.

The line is sour grapes, though. Anthropic is enormously influential, and so all of its ideas receive a great deal of attention, whereas the rest of us typically get in the spotlight only by doing well on some benchmark (e.g., AxBench).

This is an obscure reference to the famous application of SAEs to Golden Gate Claude by Anthropic. The gist of the work is that, by amplifying SAE features related to the Golden Gate Bridge and then using the reconstructed output in place of the input in the larger model, the team could get Claude to mention the Golden Gate Bridge (and related concepts) in all of its outputs. This inspired a lot of subsequent work on steering. The Golden Gate Bridge is roughly north(west) of the Anthropic headquarters, and “steer northly” is meant to invoke this – but it is obscure, as I said.

“Claim the V” refers to claiming a victory, in the sense that this was supposed to be a clear win for SAEs. However, the original work does not compare against robust baselines (e.g., simple prompting) or other state-of-the-art interp methods. The central argument of the AxBench paper is that SAEs are not even a competitive method for this kind of steering, and no internal (i.e., representation-based) methods do better than prompting at the basic version of the task.

Golden Gate Claude was nonetheless the moment that led the entire field to go all-in on SAEs as the primary tool for interp. The “irrationally” line here is meant to say two things: narrowly, that the win here was not declared on a rational basis of evidence; and broadly, that the field’s reaction to SAEs was not rational in that it was not based on a clear assessment of evidence or a systematic study of salient alternatives.

These lines refer primarily to a December 1, 2025, post, led by Neel Nanda, arguing, in effect, that the original research plan for mech-interp had failed to advance its near-term goals sufficiently, and that it was time to shift to “pragmatic interp”: A pragmatic vision for interpretability.

For me, this is a complicated judgment. The primary target of the article is the goal of using interp methods to completely reverse engineer real-world, large-scale LLMs. This is an extremely ambitious goal, and these authors only gave it about 4 years. One might expect it to take 40 years, or even 400. On the other hand, if the motivation for interp is safety, then it does seem much more effective to focus on smaller wins that will advance safety goals.

In any case, though, this post (along with an earlier one from DeepMind) pushed SAEs into further decline. In addition – and even more unfortunately from my perspective – it also led a lot of people in my own life to become more skeptical of the entire interp project.

This is a second reference to transcoders, a newer sparse interp method. The foundational proposal for using transcoders to do interp work is Dunefsky et al. 2024, and the approach became very prominent when Anthropic researchers generalized the idea to crosscoders (Lindsey et al. 2024).

It’s not really fair to say that they are “the same”. Transcoders attempt to learn sparse, interpretable versions of model components, whereas SAEs unpack a single vector of activations into a sparse representation. Another perspective on this is that transcoders learn a sparse approximation of a computation, whereas SAEs learn a sparse approximation of a representation. However, as Lindsey et al. 2024 show, it is possible to express all of these ideas in a unified way:

1. Recall that an SAE seeks to reconstruct its input vector. A transcoder seeks to reconstruct the output of a model component based on its input. The parameterization has the same form in both cases.

2. The crosscoder generalizes the SAE/transcoder encoder layer to allow for summing over multiple layers:

   h = f(Σ_l W^l_Ea^l(x) + b_E)

   where l is a layer, W^l is the encoder weight matrix for l, and a^l(x) gives layer l activations for input x. If we sum over just one l, then this reduces to a standard SAE/transcoder encoder. For the decoder, we can try to reconstruct the summed layer (roughly like an SAE) or a different layer (like a transcoder). Lindsey et al. 2024 explore many variants of this idea and relate them to different hypotheses about feature representation and information flow.

I would argue that the underlying philosophy is shared: that a sparse version of the target model can be faithful enough to the target model to serve as an explanatory basis for that model. From many perspectives, the causal methods lauded at the end of this verse and dissed in the next verse take a substantially different approach. However, as we will see when we get into that verse, there are many points of conceptual overlap, so claims about how they differ tend to feel exaggerated once one starts thinking about them as flexible tools.

This is a final but important dig at SAEs: the usual path to using them as interp tools does not end with the hidden SAE representation h. That vector itself is large and somewhat unwieldy from the perspective of human comprehension. By far the dominant response to this, pioneered by Bills et al. 2023, is to employ auto-interp methods: prompt a language model to describe (a textual summary of) the vector of interest.

Of course, at this point, we are outside the bounds of the mathematical framework of SAEs, and any guarantees we thought we had about reconstruction faithfulness are gone as well: an LLM is simply generating text. We can look to see how reliably the generated texts allow another LLM to regenerate the initial activations, but even doing perfectly at this task is not a guarantee of faithfulness in a human sense. Rather, it just shows that text is sufficient in some sense for reidentifying the input pattern.

This is a very basic description of an SAE at the level of its core computation graph. A fuller version would look like this:

h = f(W_E(x – b_D) + b_E)

x’ = W_Dh + b_D

The first line gives the encoder and the second the decoder. Here, x ∈ ℝ^d is the input activation vector. The encoder weights are W_E ∈ ℝ^m×d and bias term b_E ∈ ℝ^m, where m is the dimensionality of the sparse hidden layer. The decoder weights are W_D ∈ ℝ^d×m and bias term b_D ∈ ℝ^d. Finally, f is an activation function. A typical value for f is ReLU (as the chorus says). In the modern era, f is usually chosen to encourage sparsity.

The verse’s emphasis on W_D, standing in for the decoder, is a remnant of the fact that the verse initially focused on steering, where the decoder plays a primary role. In retrospect, it would have been better to find a way to center the sparse hidden representation, but I have not found a satisfying formulation yet. The following is a bit long but might be close:

Encode x to a big sparse rep, Decode back to x-prime, one step, Seems like there’s tons to see, This whole thing just hypnotized me.

Another reference to steering applications of SAEs (see also the “Steer northly” line above).

This is a reference to Distributed Alignment Search, a causal method for finding features as directions in activation space. DAS-like maneuvers get dissed in the next verse.

These lines describe basic interventions one might perform on an LLM’s representations to try to learn something about its abstract causal structure. The two methods mentioned are zeroing out (“knocking out”, “ablating”) representations and replacing representation values with the average of the values over some set of inputs to the target model. Both of these are, intuitively, attempts to remove specific components. Of these two, it seems to me that the mean is more defensible; neither operation actually removes, but rather only replaces with specific values, and the mean is at least a coherent default to choose (Wang et al. 2023).

Ignoring these technical hiccups, we can connect these interventions with a causal story. For example, suppose our hypothesis is that the representations we intervene on with these operations do not matter for the overall functioning of the network. This is akin to claiming that they correspond to irrelevant high-level variables, and the prediction is that their “removal” will not change the input–output behavior of the overall model. Conversely, if we target representations that we hypothesize to be necessary for a specific capability, then these interventions should destroy that capability.

These lines begin to describe interchange interventions (often called “activation patching”), which involve replacing representations with the values they take on when the model processes a different example. As far as I know, Vig et al. 2020 were the first to develop this technique for neural networks. Atticus Geiger independently rediscovered the technique, building on Beckers and Halpern 2018, and we published Geiger et al. 2020 just after Vig et al. (which we knew about by then and so cite extensively). Meng et al. 2022 refer to their operation as “causal tracing”, and Wang et al. 2023 seem to have coined “activation patching”.

In a bit more detail: suppose we are studying a model M. We have M process the input sequence “2 * (3 + 5) =” and observe that it correctly outputs “16”. Now we hypothesize that one of the internal representations, call it h, stores the sum of “3 + 5” as an intermediate variable. To test this hypothesis, we have M process “3 * (2 + 1) = ” and observe that it correctly outputs “9”. Now the interchange intervention: replace the values for h in the first example (called the “base”) with the values of h in the second example (the “source”). If our hypothesis is correct, this will cause the intervened model to output “6”, because we have in effect turned it into a model computing 2 * (2 + 1). If this works, we have a bit of evidence for our hypothesis about what h stores. If it works for all relevant inputs, then we will have a stronger case for our hypothesis.

These operations often reveal extensive, robust modular structure in the internal representations of modern foundation models. I take this to be an explanation for why they are able to solve very hard generalization problems. But then come the disses. The first diss is that we often don’t know which representation will contain the information of interest, and so we do these interventions at all conceivable representations – a decidedly “brute force” search. Modern models have many internal states, and the more we search through them, the greater the risk that we will happen upon spurious structure.

I can’t resist pointing out that SAE advocates would surely like to train an SAE on every representation in the model. However, they have to be selective because training SAEs is so expensive.

This line continues the previous line’s diss. The emerging criticism is that causal methods are unconstrained, or used in unconstrained ways, and that this might be leading us to find things that aren’t really there in some sense.

The idea of “learning subspaces” can be read as a reference to any intervention method that is going to try to learn to find features as directions in activation space. (Notice that this is the same starting point as we had for SAEs in verse 1, line 1.) The Distributed Alignment Search (DAS) method of Geiger et al. 2023 (mentioned in the first chorus) is a prominent example. DAS takes direct inspiration from Smolensky 1986, who hypothesizes that neural networks store information as “patterns”, i.e., linear combinations of unit vectors.

The goal of DAS is to perform soft interventions (as opposed to the hard interventions described in this annotation). Intuitively, this happens by rotating the base and source representations using a rotation matrix, editing the base features to be those of the source, and then rotating this edited representation back into the original basis. This edited vector is added into the original base, which edits the target neurons while preserving all others.

DAS seeks to learn a rotation matrix R that maximizes the accuracy of these soft interventions. (All parameters of the target model are kept fixed.) This is generally done in a supervised manner. To make this concrete, let’s continue the math example from above:

1. Gather a dataset of examples like “2 * (3 + 5) = 16” and “3 * (2 + 1) = 9”.
2. Define a causal model (e.g., a computer program) that says what the correct label should be for each intervention. For example, if we intervene on the intermediate value (3 + 5) in the example “2 * (3 + 5) = 16” and change it to 3, then the result is 6.
3. Perform a DAS intervention on the target neural model and get the result. If the result does not match the label defined in step 2, then you have an error signal that you can use to update R.

In essence, we are learning a classifier in this example, but with two twists: we are using counterfactuals, and we are trying only to improve our ability to do interventions on the target model successfully.

Recall that, for SAEs, the focus of the interpretability effort is the sparse hidden vector h. For DAS, the focus is the rotation matrix R. In the context of the DAS model, this defines the learned intervention.

As with classical interventions, it is common to fit DAS models on essentially every activation vector in the target model, assess each one on test data, and use that test-set performance as an estimate for the extent to which the target feature is present. So we are “learnin’ subspaces, all over the places (c’mon)”.

What is the best way to perform causal interventions? The field is in an exploratory mode, and so lots of variants are being tried. This line expresses skepticism that we will resolve these questions, because the conceptual space of interventions is so complex. Geiger et al. 2025 provide some guideposts by classifying different methods in terms of the intervention algebras they define, but this is just one step forward in a vast space.

I do think that intervention algebras are a powerful mode of thinking. The idea is that an intervention method should satisfy two basic properties: commutativity (if two interventions target different variables, the order in which they are applied doesn’t matter) and left-annihilativity (if two interventions target the same variable, only the second one has an effect). Intervention methods that are intuitively well-behaved do satisfy these requirements, and there may be further axiomatic requirements we could impose to further refine our ideas about what count as useful interventions for interp.

Why are we doing this at all? As I noted above, these methods have found high-level concepts in these networks, and this is enabling us to control them more reliably. However, we might still worry that this is just imposing our own human perspective. What if this is the wrong way to think about neural network representations entirely? This would mean that our current insights, however useful for control, do not reflect how these networks actually work.

These lines begin to instill the doubt. Are learned causal intervention methods too powerful? If so, can they be tamed?

This line shouts out what might be the most significant risk for advanced causal intervention methods: their capacity to find structure in randomness. This is a version of the faithfulness issue that is so central to the theory of SAEs (see the instinct annotation above).

This might be unavoidable for any method that is seeking to explain very large neural networks. (SAEs are not immune, for example.) The reason is that neural networks have so much representational power. As a result, as the size of the activation vectors grows, it becomes a near certainty that a sufficiently powerful interp method will find directions/subspaces that correlate with almost any target. The features are truly represented, but they do not feel real or significant to us.

For this reason, we have done comparisons against random baselines ever since our earliest work with DAS. For examples of these comparisons, see Geiger et al. 2023, section 4.4; Wu et al. 2023, section 4.7; and the selectivity metrics of Arora et al. 2024. My takeaway is that the best way to combat spurious results in this space is to perform experiments that require the interpretability method’s solutions to generalize to new cases. The structure we find in random networks fails to generalize in any meaningful way. (For excellent discussion, see Geiger, Harding, and Icard 2025.)

This is not a panacea, though. Imagine we train the interpretability method on all relevant examples for a given feature F and find that the method perfectly identifies F. This will give us the strongest guarantees that we have found causal structure, but it will mean that we no longer have an empirical procedure (hard test cases) for distinguishing real from spurious structure.

Similar issues can arise for non-random networks. We had a short but productive debate about this issue: Makelov et al. 2023 contend that DAS can find illusory structure, and we (Wu et al. 2024) responded to say that the structure is real. In my view, the structure DAS finds in these examples is mathematically real; the substantive disagreement is about whether it matters, which is a difficult philosophical question. If the intervention opens up a computational pathway that cannot be obtained by any input, is that pathway real? It seems to me that it is, but I have to grant that it is not clear whether it is structure that could be relevant. If impossible cosmic circumstances (and only those) could enable you to sing beautiful operatic arias, would you care?

This issue is compounded by the power of our methods. Sutter et al. 2025 prove that very powerful non-linear interp methods will be able to find arbitrary structure in a network. This seems to cut very deep, since merely asking for a hard generalization test will not overcome it. This issue is, in many ways, prefigured by Geiger et al.’s (2025) study of interventionals, which are functions that look like intervention methods but actually have the capacity to change the target network in arbitrary ways that clearly are not faithful.

Hmm, I might be showing my biases here: it is good that we are talking about direct effects, since that is a signal that we are finally going beyond correlational methods to consider causal claims!

This too is not a diss, but rather just shouts out a paper I was involved with: Grant et al. 2026. The central observation of the paper is that intervened representations always diverge from the original in non-trivial ways. We discuss the nature of these divergences and try to determine where they are helpful and where they are harmless, and we develop a variant of DAS that reduces divergence while still achieving intervention effects.

I should add that divergence is not unique to causal methods. SAEs also create divergence when used causally: if you run the SAE on input x to obtain reconstructed output x’, and then you use x’ (or a modified variant thereof) in place of x in the original network (say, to steer the model in some way), then you will have created a divergent representation via your intervention. This is discussed in this annotation under the rubric of faithfulness.

I just find this line funny: “do respects” is a reference to the Do-calculus (Judea Pearl’s calculus for reasoning about interventions), and “do calculus” shouts this out again in a way that sounds like a bit of advice to researchers.

In truth, AI researchers do more than enough calculus of a different kind (the differential kind that powers gradient descent). It’s one of the rarely explored near-axioms of AI that gradient-based learning is the best kind of learning.

A callback to the first verse. As discussed in this annotation, transcoders are a kind of next-generation version of the SAE vision for using sparse networks to do interp. If causal interp researchers thought it would suffice to outdo SAEs, they should think again, because now they have to contend with these newer ideas!

This is a real diss – a real concern for causal methods that use interchange interventions. To perform those operations (described in this annotation and this one), we need to have aligned pairs of examples. This can lead to lots of idealizations and compromises. Consider, for example, the mathematical example involving pairs like “2 * (3 + 5) = 16” and “3 * (2 + 1) = 9”. If we go searching for the “inner sum” variable, we will want to use closely aligned examples like this, and figuring out what to do with even closely related examples like “(1 + 2) * (3 + 5) = 24” will be tricky. In the end, we might stack the deck in favor of the hypothesis being tested.

By contrast, correlational methods like SAEs and transcoders do not require such aligned data, and so it might be said that they can avoid these artificialities. They have their own issues, as discussed throughout the first verse of the song, but this is one they can avoid.

This line seeks to convey a unifying insight: essentially all methods under consideration in this song (as well as other leading methods in the field) can be used causally or given a causal construal. Some examples:

1. Probing is a classic correlational method: you fit a supervised probe model for task T on the internal representations h of a target model, and use the performance of the probe as an estimate of how much h contains information about T. This alone can’t tell you whether the T capability has anything to do with the model’s behavior. However, the probe itself will have parameters that can be used to steer the model via an intervention. Where the probe is a log-linear classifier, its weight vector can be used to define an intervention on h (Arora et al. 2024).
2. SAEs are correlational when we think about h, but if we use the reconstructed output in place of the input, we are performing a causal intervention. If we first modulate the features in h before using the decoder to get the new output, we are steering. Steering can be construed as the basis for testing causal claims. The same holds for transcoders and crosscoders.
3. Even attribution methods often have straightforward causal interpretations. For example, Integrated Gradients estimates the individual causal effect of a neuron with respect to the chosen input and output (Geiger et al. 2021).

These examples show that, ultimately, it is unproductive to classify methods by whether they are causal or not. The substantive question is what conclusions they support when used in a particular way.

This diss has to refer to the labor involved in creating counterfactual pairs. From other perspectives, causal methods are faster than SAEs.

Another diss related to the “random” line above.

This is one way to parameterize DAS. Where s ∈ ℝⁿ is the source representation, b ∈ ℝⁿ is the base, and R ∈ ℝ^d×n is a matrix with orthonormal rows, with d giving the dimensionality of the intervention:

b := b + R^T(Rs - Rb)

This is a soft intervention in the sense that it defines a direction in the feature space and seeks to intervene only on that direction, preserving the other dimensions of b.

It is common to refer to R as a rotation matrix. In the above formulation, it is not quite that (it is not a square matrix). If R is a full rotation matrix, then an additional projection matrix is defined to select the d-dimensional intervention subspace. The two formulations are identical; the low-rank R here combines the rotation and the selection into one matrix.

This is another way to parameterize DAS:

b := (I - R^TR)b + (R^TR)s

where, as above, s is the source representation, b is the base, and R is the learned matrix with orthonormal rows (the “rotation matrix”). I is the identity matrix. R^TR is sometimes called a “Gram matrix”, but this might be somewhat annoying to experts, since it is also projecting onto the d-dimensional intervention subspace.