This result looks pretty stunning.
1. Train against reward hacking classifier -> learn to hack evasively
2. Apply gradient routing—i.e. localize gradient updates from reward-hacking episodes to parameters that are knocked out in deployment -> no hacking, even evasively
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
