Model Lifecycle Management & Change Control · Pipeline Fine-Tuning
Detect reward hacking in RLHF
Verify that RLHF training stages include automated detection of reward hacking or reward model over-optimization.
OWASP AISVS v1.01, requirement 3.5.2
In plain English
In RLHF the model is trained to score well with a reward model. Reward hacking is when it learns to please the reward model without getting better, for example by flattering, padding or exploiting a quirk in the scorer. Detect this automatically during training. It is a Level 3 control for teams running their own RLHF.
What good looks like
- Each training stage tracks signals of over-optimisation: reward rising while held-out quality stalls or falls, or outputs drifting far from the reference model.
- Runs showing those signs stop or pause for review automatically.
- People review output samples at set points in the run.
- The held-out evaluation uses a judge the model was not trained against.
How teams usually do it
- Track the KL divergence from the reference model alongside the reward, and stop early when it crosses a limit.
- Evaluate checkpoints with independent measures: a second reward model, a held-out human-rated set or task benchmarks.
- Use an ensemble of reward models and treat strong disagreement between them as a warning.
Suggested evidence
- Logs or dashboards from training runs, showing reward and independent metrics together.
- The stopping rules and an example of one firing.
- Records of human sample reviews.
Common mistakes
- Judging the run by the reward score alone.
- Using the same reward model for training and for the final evaluation.
- Reviewing samples only at the end, after days of compute have gone into a degenerate model.
Not sure which requirements apply to your system? Take the assessment, or browse every guide in this chapter.
Requirement text from the OWASP AI Security Verification Standard, licensed under CC BY-SA 4.0. This guidance is shared under the same licence.