LLM Inference Interview Questions #19 - The 10x Sampling Trap

Why throwing more inference budget at a reasoning task quietly yields ninety thousand identical wrong answers, and how decomposing your flat curve into coverage vs. selection saves you from a six-fig.

You’re in a Senior AI Engineer interview at DeepMind and the interviewer asks:

“You 10x’d your sampling budget on a hard reasoning task and the solve rate barely moved, even though the paper’s log-linear scaling curve says it should have. What’s the first thing you measure?”

Don’t say: “The base model isn’t strong enough, we need a bigger model.”

That’s the answer that ends the interview. You just proposed a six-figure fix without measuring anything.

Here’s the reality: there are two completely different failure modes hiding behind one flat curve, and you can separate them in an afternoon.

Measure this first: pass@k on the full sample pool vs. the score of the subset you actually ship.

Two very different diagnoses:

• Coverage is still growing, but your solve rate isn’t. Then the correct solution IS in your sample pool and your selection stage is throwing it away. Your bottleneck isn’t generation, it’s the filter and rank layer. This is why AlphaCode’s pass@k topped 40% while its 10@k sat at 30%. The extra 10% was generated and then discarded. More samples cannot fix a broken scorer, it just gives it more chances to be wrong.

• Coverage itself has flattened. Then you hit coverage saturation, and the reason is almost never “the model is weak.” It’s that your 10x more samples aren’t 10x more distinct samples. You paid for volume and got redundancy.

The metric nobody instruments: effective sample diversity. Cluster your generations by behavioral equivalence and plot unique cluster count against sample budget, not raw sample count. If clusters plateau at 10k while you’re paying for 100k, you’ve been buying the same wrong answer ninety thousand times.

You’re not widening the search. You’re re-rolling the same die and hoping.

The fix is diversity engineering, not scale: randomize temperature and metadata, and generate from a family of differently-tuned variants rather than one checkpoint. That’s exactly the move AlphaCode 2 made, and it matched AlphaCode’s million-sample performance at roughly 100 samples.

---

The answer that gets you hired: log-linear scaling only holds if each new sample is a genuinely new hypothesis. So before you touch the model, decompose the flat curve into coverage growth and selection loss. One of those is a diversity problem, the other is a reward-model problem, and neither is solved by buying more GPUs.

#AIEngineering #LLM #MachineLearning #TestTimeCompute #AIAgents #MLOps #SystemDesign

---