RAG Interview Questions #15 - The Ambiguity Relocation Trap
How collapsing queries into a single confident guess tricks your evals while destroying intent fidelity, and why the fix requires rewrite candor and late-stage arbitration.
You’re in a Senior ML Engineer interview at Anthropic and the interviewer asks:
“You added an LLM query rewriter to fix ambiguous queries. Recall went up in your offline eval. So why did production accuracy quietly drop three weeks later?”
Don’t say: “The rewriter must have a bug” or “We need a bigger model.”
Too shallow. You’re treating a systemic failure like a code defect.
Here’s what’s actually happening. 👇
Your rewriter isn’t broken. It’s confidently wrong in ways your eval was never built to see.
An LLM rewriter resolving “when did he start the company?” has to guess the missing referent. And a guess is a silent decision:
Referent hijacking. The user meant PayPal. The rewriter picked Tesla, because Tesla is more frequent in its training distribution. The query is now grammatically perfect, retrieves flawlessly, and answers the wrong question. No error. No exception. Just a clean, wrong answer.
Intent narrowing. Rewriting collapses a broad query into one crisp interpretation. You gained recall on the interpretation you picked, and silently destroyed recall on every interpretation you didn’t.
The eval blind spot. Your offline set scores rewrites against a known ground-truth intent. But the entire failure mode is the ambiguity of intent. Your eval already resolved the thing production has to guess. You’re grading the test with the answer key taped to the page.
The insider framing:
Query rewriting doesn’t remove ambiguity. It relocates it, from a visible retrieval miss (which you’d catch) to an invisible intent mismatch (which you won’t). You’ve traded a loud failure for a silent one.
The fix isn’t a better rewriter. It’s rewrite candor:
