TL;DR

• We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
• The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
• A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
• WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
• Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no chain of thought. This is bad for CoT monitorability and makes interpretability essential to actually understanding what is going on. A key goal of interpretability is to understand intermediate variables that a model uses to compute its answers. The intermediate representations that models store in their global workspaces contain useful information that can help us decode their intentions, beliefs, algorithms, and thought processes. However, we don’t currently have a good way to measure whether an interpretability tool recovers such variables correctly. We made a benchmark to test how well an activation-to-text tool can read the contents of a model’s global workspace. We think overall performance on WorkspaceBench is a good proxy for practical utility in model auditing, and we hope that new interpretability tools can be evaluated on WorkspaceBench upon release to help determine their capabilities.

Why has no one made a WorkspaceBench before?

Interpretability tools are hard to evaluate, in part because a tool can score well without reading the workspace. For example, a tool that simply inverts activations back into the input text (text inversion) lets an agent infer an intermediate from the prompt rather than from the activations. Our benchmark doesn't fully rule out shortcuts like this, but we address three problems that make evaluating interpretability tools difficult:

1. There’s no ground truth: we don’t know what is actually in a model’s global workspace, so it’s hard to tell what should count as surfacing workspace content.
   1. How we fix this: We design tasks such that we can be reasonably sure the model needs to use a certain intermediate to compute the answer. We also tailor questions to the model’s capability levels to ensure it can consistently deliver the correct answer.
      1. For some evals (e.g. association and directed modulation) we can’t be absolutely sure the model is thinking about these concepts. However, many tools seem to be able to surface this type of information, which suggests these representations are often held in the workspace. Therefore, we do not necessarily insist that it is possible to saturate the benchmark on these tasks, but higher scores are still directly indicative of a tool’s capabilities
2. Models often don’t have sophisticated enough computations to have strong representations of intermediates
   1. How we fix this: Recent open source models are increasingly capable of having strong workspace representations, so we use Qwen3.6-27B to test the various interpretability tools on surfacing interesting intermediates.
3. It is unclear how to weigh the pros and cons of different activation-to-text methods as workspace readers: for example, J-lens is single token but reliable, while NLAs are expressive but prone to confabulation.
   1. How we fix this: We measure WorkspaceBench accuracy scores vs hallucination rates to study such tradeoffs in different tools

Background

We evaluate the following different activation-to-text methods on how well they extract intermediate variables in Qwen-3.6-27B’s workspace:

• Single-Token Readers: methods below take in a single activation to give back a ranked list of top tokens in the model’s vocabulary.
  • Logit lens: a training-free method that applies the unembedding matrix directly to a model’s residual stream.
  • J-lens: uses the Jacobian to capture the mean linearized effect of an activation to produce tokens in the vocabulary. Requires fitting a linear map from the final layer to the readout layer via backprop and averaging the Jacobian over contexts/positions.
  • R-lens: a drop-in replacement for J-Lens that uses layer-wise relevance propagation (LRP) to surface concepts earlier than J-Lens in a model’s workspace. Similar cost to J-Lens.
  • Tuned lens: a variant of logit lens that learns an affine trans

TL;DR

• We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass.
• The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools.
• A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval.
• WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks.
• Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here.

Introduction

Astra can do a concerning amount with no chain of thought. This is bad for CoT monitorability and makes interpretability essential to actually understanding what is going on. A key goal of interpretability is to understand intermediate variables that a model uses to compute its answers. The intermediate representations that models store in their global workspaces contain useful information that can help us decode their intentions, beliefs, algorithms, and thought processes. However, we don’t currently have a good way to measure whether an interpretability tool recovers such variables correctly. We made a benchmark to test how well an activation-to-text tool can read the contents of a model’s global workspace. We think overall performance on WorkspaceBench is a good proxy for practical utility in model auditing, and we hope that new interpretability tools can be evaluated on WorkspaceBench upon release to help determine their capabilities.

Why has no one made a WorkspaceBench before?

Interpretability tools are hard to evaluate, in part because a tool can score well without reading the workspace. For example, a tool that simply inverts activations back into the input text (text inversion) lets an agent infer an intermediate from the prompt rather than from the activations. Our benchmark doesn't fully rule out shortcuts like this, but we address three problems that make evaluating interpretability tools difficult:

1. There’s no ground truth: we don’t know what is actually in a model’s global workspace, so it’s hard to tell what should count as surfacing workspace content.
   1. How we fix this: We design tasks such that we can be reasonably sure the model needs to use a certain intermediate to compute the answer. We also tailor questions to the model’s capability levels to ensure it can consistently deliver the correct answer.
      1. For some evals (e.g. association and directed modulation) we can’t be absolutely sure the model is thinking about these concepts. However, many tools seem to be able to surface this type of information, which suggests these representations are often held in the workspace. Therefore, we do not necessarily insist that it is possible to saturate the benchmark on these tasks, but higher scores are still directly indicative of a tool’s capabilities
2. Models often don’t have sophisticated enough computations to have strong representations of intermediates
   1. How we fix this: Recent open source models are increasingly capable of having strong workspace representations, so we use Qwen3.6-27B to test the various interpretability tools on surfacing interesting intermediates.
3. It is unclear how to weigh the pros and cons of different activation-to-text methods as workspace readers: for example, J-lens is single token but reliable, while NLAs are expressive but prone to confabulation.
   1. How we fix this: We measure WorkspaceBench accuracy scores vs hallucination rates to study such tradeoffs in different tools

Background

We evaluate the following different activation-to-text methods on how well they extract intermediate variables in Qwen-3.6-27B’s workspace:

• Single-Token Readers: methods below take in a single activation to give back a ranked list of top tokens in the model’s vocabulary.
  • Logit lens: a training-free method that applies the unembedding matrix directly to a model’s residual stream.
  • J-lens: uses the Jacobian to capture the mean linearized effect of an activation to produce tokens in the vocabulary. Requires fitting a linear map from the final layer to the readout layer via backprop and averaging the Jacobian over contexts/positions.
  • R-lens: a drop-in replacement for J-Lens that uses layer-wise relevance propagation (LRP) to surface concepts earlier than J-Lens in a model’s workspace. Similar cost to J-Lens.
  • Tuned lens: a variant of logit lens that learns an affine trans