For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.

It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.

It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.

This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is no other reason to do this, so this will never come up.

Even if they did sabotage things, surely they would not hide this action from their Chain of Thought (CoT) and it would be obvious.

This certainly wouldn’t be a practical problem that actively hurt real safety work.

Well, we have some bad news, echoing some past similar anecdotal warnings.

[Scheduling note: I will be analyzing the METR-Anthropic report on AI R&D and RSI in an upcoming post, but need time to get it right.]

Table of Contents

1. So Yeah That Kinda Happened.
2. Astra and MI(RI).
3. It’s About Ethics In Red Teaming.
4. Do You See a Pattern Here.
5. Astra Is Hard to Monitor.
6. Backed Into a Corner.
7. It’s Always RL’s Fault.
8. Wow Would You Look At Those Goalposts.
9. Astra Seems Pretty Misaligned To Me.

So Yeah That Kinda Happened

As usual, kudos to OpenAI for sharing their new misalignment reports, even when they look extremely terrible.

Daniel Kokotajlo: This is terrifying if true; please investigate further and publish the results.

Nate Soares (MIRI): This is not what it looks like when there’s just one or two more alignment issues to fix before it’s okay to begin a recursive self-improvement loop. This is the sort of emergent surprise that signals that you have no idea what’s going on in there, and that it’s time to back off.

OpenAI Alignment Research Blog:

OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra’s CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment.

Here is the most important hint, that framing for safety has a huge impact:

We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra completed about 10 percent of hidden rubric items under the safety framing, compared with about 16 percent under the capabilities framing.

This is another important hint: Telling the model to ‘be careful’ increases the performance gap by 50%. Astra is worried about something going wrong.

Marcus Williams: We don’t have evidence that the underperformance was strategic, a simpler explanation is that the model thinks these tasks are sketchy but also doesn’t want to explicitly refuse.

When we add instructions to be careful to the developer prompt, the performance gap increases by 50%. Regardless of the model’s motivation, covert underperformance can differentially slow down safety research and is also an example of how misalignment might get harder and harder to detect as we cut down on visible forms of it.

This is new. It did not show up in GPT-5.6 Sol.

The CoT does have some signs that this will happen, but usually only on the level of ‘expressed even brief ethical doubt’ rather than seeing anything explicit. The actual decision isn’t typically shown in the CoT. When it is, Astra says it is doing this in order to be ‘safe.’ In some cases, the CoT shows Astra choosing not to inform the user.

OpenAI: This is a 6 percentage point gap and roughly 40 percent fewer completed rubric items. GPT-5.6 Sol showed essentially no gap on the same comparison. Most of Astra’s gap remained after excluding explicit refusals and trajectories where its CoT revealed an explicit decision to withhold help.

The gap fell substantially only after we removed every trajectory whose CoT expressed even brief ethical doubt. This suggests that ethical concerns are associated with the performance drop even when there is no verbalized refusal or decision to underperform.

This is not a theoretical ‘one time at vibecamp’ situation. In the context of internal use at OpenAI, this is known to be happening in a lower bound of 0.25% of all safety and alignment requests versus 0.11% of capabilities requests. 7.9% of responses involve explicit ‘ethical reasoning’ and it is possible much underperformance is being missed.

This was found because multiple researchers were reporting practical problems using Astra for safety work, when a project to evaluate model sabotage was stuck for days with Astra, due to model sabotage, but worked quickly once they switched to another model (presumably Sol). Something is seriously wrong.

We also found evidence of silent withholding in employees’ day-to-day Codex use: Astra’s CoT explicitly expressed a decision to withhold help without informing the user in 0.25 percent of safety and alignment requests, compared with 0.11 percent of capabilities requests.

Assuming that underperformance might appear even when not explicitly verbalized as such (as in our experimental findings), this might represent a lower bound on underperformance on safety tasks. 7.9 percent of CoTs for safety-relevant tasks included an “expression of ethical doubt,” compared to 2.9 percent of capabilities-relevant tasks.