RAG Interview Questions #16 - The Green Dashboard Paradox

The hidden danger of AI vetting AI: why your evaluator gets more confident as your output tanks, and the architectural fix needed to anchor your system back to reality.

You’re in a Staff ML Engineer interview at Anthropic and the interviewer asks:

“Your Corrective-RAG system has an AI evaluator scoring its own retrievals. Six months in, answer quality is degrading, but your dashboards are green. What’s happening, and how would you have prevented it on day one?”

Don’t say: “The evaluator threshold needs retuning.”

Here’s t…