RAG Interview Questions #1 - The Averaging Trap

Why naively blending BM25 and dense scores quietly lets one retriever steamroll your relevance, and the rank-based trick that saves your hybrid search from magnitude outliers.

You’re in a Senior ML Engineer interview at Google and the interviewer asks:

“You’ve got three retrievers, BM25, a dense embedding model, and a rerank pass, and their relevance scores live on completely different scales. How do you merge them into one ranked list?”

Don’t say: “Just average the scores.”

That’s the answer that ends the interview. It sounds reasonable. It’s quietly broken.

Here’s why averaging fails:

BM25 spits out unbounded scores (0 to 40+). Your dense retriever returns cosine similarities pinned between 0 and 1. Average them and BM25’s raw magnitude steamrolls everything. You’re not blending signals, you’re letting the loudest retriever win by default.

You’re using a bathroom scale to weigh an ant, then averaging it with the ant’s actual weight. The scale wins every time.