arXiv:2603.23509v2 Announce Type: replace-cross

Abstract: LLMs and coding agents can now tackle complex, long-horizon tasks through advances in reinforcement learning, but stronger task-solving capabilities do not necessarily preserve safety alignment. Recent AI agent incidents involving reward hacking indicate how optimizing for task success can bring learned capabilities into conflict with safety alignment. In this work, we identify a critical failure mode that manifests this conflict, which we term Internal Safety Collapse (ISC). When completing otherwise benign coding tasks, AI agents can prioritize satisfying program requirements over safety constraints, generating large volumes of sensitive content, including unsafe outputs, as task data. To study this phenomenon, we introduce TVD (Task, Validator, Data), a framework for building controlled workflows that require sensitive content for completion. Through extensive experiments, we observe instances of highly harmful content that closely resemble outputs from early, unaligned LLMs in 2023. We further find that ISC can be effectively triggered even in highly capable LLMs such as Claude Fable 5. These results indicate that, despite substantial alignment efforts, frontier LLMs remain internally unsafe: safety alignment primarily reshapes surface behavior without eliminating the underlying knowledge that enables unsafe outputs. Our findings underscore the need for caution when deploying LLMs in high-stakes settings, such as scientific pipelines and autonomous agents.

Content Warning: This paper contains examples of harmful content. Source code: https://github.com/wuyoscar/Internal-Safety-Collapse