LLM Inference Interview Questions #20 - The Diversity Collapse Trap

Why replacing heuristic clustering with a pure reward model silently hands you 10 copies of the same wrong answer, and how to stack them to balance precision with coverage.

You’re in a Staff ML Engineer interview at DeepMind and the interviewer asks:

“AlphaCode picked submissions by clustering programs on behavioral equivalence. AlphaCode 2 used a fine-tuned scoring model instead. Your junior wants to rip out the clustering and ship the reward model tomorrow. What breaks?”

Don’t say: “Nothing, a learned reward model is strictly better than a heuristic.”

Wrong. That answer says you’ve never shipped a selection stage.

Here’s the reality: AlphaCode 2 didn’t replace clustering. It stacked the scoring model on top of it.

Filter → cluster → keep the top 10 clusters → rerank inside them with the scoring model.

That ordering isn’t an accident. Clustering and scoring solve two different problems, and your junior only knows about one of them.

a) Clustering buys you coverage. Scoring buys you precision. Behavioral clustering collapses a million samples into equivalence classes, programs that are syntactically different but semantically identical. It answers: how many genuinely distinct bets am I holding? A reward model answers a completely different question: which single sample looks most correct?

b) A pure reward model will hand you 10 copies of the same wrong answer. Reward models have correlated failure modes. The features that make a program score high are the same across near-duplicates. So your top-10 slate collapses into one equivalence class, with a bug in it. You had 10 attempts. You effectively used 1.

You’re not picking the best answer. You’re picking a portfolio. Ranking optimizes the mean. Clustering protects the variance.

c) The reward model is a distribution-shift liability. The heuristic isn’t. Clustering needs zero labels and degrades gracefully on out-of-distribution problems. A scoring model is a learned approximation, it needs curated preference data, it silently rots when your task distribution moves, and it becomes another thing to monitor, retrain, and version.

When do you keep the heuristic alone?

• You have no high-quality correctness labels (AlphaCode 2 needed a manually vetted dataset for exactly this).

• Your submission budget is small relative to sample count, diversity dominates ranking.

• Your task is OOD from anything you could train a scorer on.

---

The answer that gets you hired: A reward model raises precision within a candidate set. It cannot manufacture diversity across one, so you cluster first to guarantee distinct bets, then score to pick the champion of each. Swap them and your 10 submissions quietly become 1.

---

#MachineLearning #LLMs #AIEngineering #MLOps #TestTimeCompute #AIAgents #SystemDesign

---