LLM Inference Interview Questions #5 - The Verifier Trap
Why relying on pass/fail tests for agent evaluation doesn't remove the critic, it just hides it, and the "test the tests" regime decision that elite teams use instead.
You’re in an AI Agents Engineer interview at Anthropic and the interviewer asks:
“You want to evaluate whether your coding agent actually solved a task. A teammate says: skip the learned critic, just train a model to generate unit tests and let pass/fail decide. It’s system-agnostic and elegant. Why isn’t this the default?”
Don’t say: “Great idea, tests are ground truth, so pass = solved.”
You just assumed the hard part away.
Here’s the reality: you didn’t remove the critic. You moved it into the test generator, and that generator is now your ground truth. So the real question isn’t “do the tests pass?” It’s “who verifies the verifier?”
Where it breaks down in production:
