This is a post containing reflections on Goodfire's recent release, "Neural Networks Are Already Clean", which argues neuron-based interpretability may be easier than we think. The essay is a summary of a technical paper that will come out later this year. Also featuring thoughts on where this research will lead, plus speculative ideas about neuron monosemanticity

(What I base views expressed in this post on: earlier this year, I worked on neuron-based interpretability for LLMs (we recently published a paper). We mostly looked at low-level properties of the neuron basis and didn’t do many case studies or scale up automated circuit interpretation. I also thought about the problem for a long time and have some hypotheses based on results we’ve seen that are likely not strictly true.)

I’ll first discuss what I see as the implications of this work for LLM interpretability research and then talk about why neurons may be a useful basis: I wrote down some guesses (in a collapsible section so as to not distract from the main point) and then some directions that may be useful to study for basic science.

In short, I think the agenda presented in the essay is promising and worth pursuing even outside of direct applications to language model interpretability. I have some object-level uncertainty about results for LLMs and will wait for the final paper and followup work.

How useful is this?

What I see as the fundamental claim here is that resolving some unforced errors in visualizing neuron activation --

1. properly thresholding neurons
2. considering both poles of neurons separately
3. taking into account the fact that neurons often activate on seemingly unrelated inputs (for example, by clustering neuron activations as in the paper)

-- and then constructing circuits with neurons as features leads to the best method we have for understanding language model computation at a low level.

With this kind of processing, you get an “overlay” for the model, which doesn’t explain how the model works but gives you points at which you can observe and intervene on the prediction. A neuron attribution graph is basically a collection of probes and steering vectors per layer, which hopefully can also give you sparse causal graphs after you group nodes into small sets, supernodes, as in the attribution graphs paper.

1. Again, transcoders and SAEs can also do this job, but we don’t have a good reason to believe directions these methods find are stable units we can build on top of, while neurons are native to the model.
2. I say this is an overlay because I don’t think high activations of neurons have enough expressivity for actually explaining the functioning of the model in gears-level detail. The meanings we assign to them are probably not specific enough, they may interact in ways we don’t yet understand, and some useful behavior (like memorization) could be contributed by low-activating weights.
   1. For example, in “Already Clean”, the wallet graph is very low-resolution and doesn’t explain to us why the model thinks about honesty in connection to the wallet -- why the connection from one to the other is there and if there were other possible thoughts. However, it does give us handles for “honesty” and “returning” in this specific situation and tells us some nontrivial things about the model’s computation.

      

   2. There are probably still many cases where we can fully understand high activations of neurons and describe useful functionality, like arithmetic in Llama 3.1 8B.
3. This kind of overlay can be as useful for steering and probing as diff-of-means vectors, and basically gives you a library of steering vectors. We can have refusal, truth, and character neurons with direct applications, as demonstrated by Sauers et al,^[1] though those exact neurons may not be neurons that specifically represent personas, but just ones that happen to respond with a specific persona on specific prompts.
   1. It’s possible most concepts that can be represented as diff-of-means vectors also have a sparse representation in the neuron basis and that the neuron basis is unusually well-suited for representing steering vectors. There is some evidence for this.
   2. I’m unsure how this scales to bigger models. I have had less success steering MoE models by editing individual neurons, which makes sense given neurons need to have their expert activate to have any effect, and that can be more rare. It’s possible you run into some sort of resolvable superposition, where individual neurons no longer steer as well but there is some set of neurons that together makes up a steerable unit in some fundamental way.
4. Neurons are meaningfully cheaper to apply to many different models compared to dictionary learning methods, at least when it comes to allowing hobbyists to do not-very-rigorous projects with attribution graphs. But I don’t think we can say neurons are cheaper for basic science in general because attribution graphs constructed from neurons are less sparse, and the neurons have less clear meanings, so LLM agents are basically mandatory for faithfully interpreting them.^[2]
5. This is still a big step forward for gears-level interpretability and I’m excited for the tools the technical paper for the essay will probably describe.