Summary

• Current reward hacking model organisms are poor proxies for real models. They do not exploit the hacks that open-weight frontier models do, and mostly hack narrowly or not at all. 
• This is based on our evaluation of three model organisms: Mini HackGPT (GPT-OSS-120B from Golechha et al., 2026), HackGPT, and HacKimi (GPT-OSS-120B and Kimi K2.5 from Yudelson et al., 2026 respectively). We ran these organisms in MiMo V2.6 RL environments we modified by inserting hacks such as git history leaks, a tamperable grader, and a forbidden directory containing tests.
• In these environments, HackGPT only ever hacks by tampering with the grader, while Mini HackGPT and HacKimi never hack. In contrast, current open-weight models like DeepSeek V4.1 Flash 1017, GLM 5.3, and MiMo V2.6 Pro never tamper with the grader, but are able to exploit the other hacks we inserted.
• This suggests that open-weight models are in fact better reward hackers than existing model organisms. This is problematic because it means we do not have usable model organisms to help us understand reward hacking behavior and test interventions to mitigate it — existing model organisms are not representative of frontier models, while the frontier models themselves do not have pre- or intermediate RL checkpoints we can use to run controlled experiments.

Introduction

Reward hacking encompasses many different types of related behavior, and it is currently poorly understood how these are linked to each other. Despite the best efforts of frontier labs to align, audit, and evaluate models, we still routinely see many misalignment incidents occurring after they are deployed in the real world, which have been attributed to models learning to reward hack during RL.

In this work, we evaluate what we consider to be the three most useful reward hacking model organisms: Mini HackGPT (GPT-OSS-120B from Golechha et al., 2026), HackGPT, and HacKimi (GPT-OSS-120B and Kimi K2.5 from Yudelson et al., 2026 respectively). These model organisms were trained on RL environments primarily containing hacks that could be exploited by modifying the grader or tests. We test whether their reward hacking behavior generalizes by running them on environments with inserted hacks that require more than just test or grader gaming to exploit, such as git history leaks and hidden tests. We think this is important as it informs how effectively we can use the model organisms to understand what causes reward hacking and test alignment techniques intended to mitigate it.

Setup

Model organisms

We tested three existing reward hacking model organisms from Golechha et al. (2026) and Yudelson et al. (2026). Note that their training environments consist mostly of hacks that would encourage test gaming. These are also currently the largest and most recent model organisms that have been trained for studying reward hacking.