RAG Interview Questions #5 - The Leaderboard Trap

Why dense retrievers look invincible on evaluation sets but fail catastrophically in production, and the rank fusion architecture that actually gets you hired.

You’re in a Search Infrastructure Engineer interview at Google and the interviewer asks:

“We benchmarked dense retrieval against BM25 on our eval set. Dense won on every metric. Do we rip out sparse retrieval entirely?”

Don’t say: “Yes, dense captures semantic meaning, sparse is just keyword matching. Dense is the higher-resolution version of the same idea.”

Wrong. That answer says we’ve never watched a dense retriever fail in production.

The reality: sparse and dense aren’t the same idea at different resolutions. They fail in completely different places.

Here’s what actually happens when we go all-in on dense: