You’re in a Senior ML Engineer interview at Anthropic and the interviewer asks:
“We need to train a rigorous LLM-as-a-Judge, but we have absolutely zero human-labeled preference pairs. How do you procedurally generate ‘rejected’ responses that are nuanced enough to actually train a robust evaluator?”
❌ Most candidates say: “Just prompt a smaller, weaker model to write a bad answer, or randomly inject formatting errors into a good response.”
The reality is training an evaluator on obvious garbage is like training a counterfeit detector using Monopoly money. It’s simply too easy.
If your “rejected” data is just poorly written, artificially truncated, or injected with random noise, your reward model learns spurious correlations. It learns to act as a grammar checker or a word counter, rather than a reasoning engine. When deployed in production, it will completely fail against a highly articulate, confident hallucination.
To force the evaluator to learn true nuance, you need synthetic verifiable preference pairs. You don’t degrade the response, you shift the instruction.