LLM Agents Interview Questions #20 - The Reward Signal Collapse Trap

When evaluators can't reliably judge advanced reasoning, PPO doesn't refine the model, it optimizes toward human-perceived correctness instead of actual correctness.

You’re in a Senior AI Engineer interview at DeepMind and the interviewer asks:

“Your RLHF pipeline relies on top-tier medical and legal experts to score outputs. But as the model scales, your PPO updates start degrading its reasoning accuracy rather than refining it. What is breaking down, and how do you fix it?”

Most candidates say: “The PPO hyperparameters are unstable, or the reward model is overfitting. We need to add a stricter KL divergence penalty to keep the policy closer to the reference model.”

Wrong approach. They are fixing the plumbing when the water source itself is poisoned.

The reality is: You’ve hit the Human Evaluator Bottleneck.

As your model achieves expert-level capabilities, human preference data rapidly degrades into a toxic reward signal.