RAG Interview Questions #3 - The Multi-Vector Trap

Why chasing nDCG benchmark wins with ColBERT silently 30x's your index storage, and the hybrid reranking trick senior engineers use to get the exact same precision at a fraction of the footprint.

You’re in a Senior ML Engineer interview at Anthropic and the interviewer asks:

“You shipped ColBERT-style multi-vector retrieval because it won on nDCG@10. Two weeks later, p99 latency tripled and your index storage 30x’d. When is multi-vector actually worth that, and what exactly did you lose when you collapsed passages into single vectors before?”

Don’t say: “Multi-vector is more accurate because it keeps more information.”

Technically true. Strategically useless. You just described the benefit and ignored the entire reason the interviewer is asking. Wrong answer.

The reality: You didn’t buy accuracy. You bought accuracy at a storage and compute bill, and whether that trade is sane depends on what you were losing in the first place.

Here’s the real mechanism.

A single dense vector forces the model to cram an entire passage into one ~768-dim representation. That’s mean pooling, and it’s lossy by design: