You’re in a ML Engineer interview at Anthropic and the interviewer asks:
“Best-of-16 rollouts with a critic reranker moves your agent from 20% to 32% on SWE-bench. Your PM wants it shipped. What do you tell him?”
Don’t say: “It’s a 60% relative improvement, let’s ship it, we just need to budget for the extra inference.”
You just quoted a leaderboard number as if it were a product decision.
Here’s what’s actually happening.
1. The gain is log-linear, not linear. You didn’t get 12 points from 16x compute. You got ~3 points per doubling. 1→2→4→8→16. Four doublings, four roughly-constant bumps.
Which means the next 12 points cost you 256 rollouts. Then 4,096.
You’re paying exponentially for arithmetic returns. Everyone quoting the headline number is reading a log-scale chart as if it were linear.
2. Your critic is the ceiling, not your generator. Two different numbers hide inside “best-of-16”:
→ Coverage, did any of the 16 trajectories solve it? (your oracle pass@16)
→ Selection, did the critic pick that one?
The gap between them is pure verifier loss. On agent trajectories that gap is usually ugly, because your ORM is grading a 100-step trajectory from an outcome label. You’re using a bathroom scale to weigh an ant - and the ant is 200k tokens long.
If you have real unit tests, use them. A deterministic verifier beats a learned critic every time.
3. The cost isn’t 16x. It’s worse. Agent rollouts diverge after step one. Your shared prompt prefix dies immediately, so KV cache reuse collapses. And wall-clock is set by the slowest trajectory, not the average.
So: 16x tokens, near-zero cache hit rate, p99 latency on every request.
When is it actually rational? Narrow:
High value-per-task: a migration, a nightly batch, one ML experiment worth $10k
Async and offline: nobody is watching a spinner
Cheaply verifiable: tests exist, so selection is free
Low volume: hundreds of tasks, not millions
Interactive, per-user, high-QPS? Never.
The answer that gets you hired: best-of-N buys leaderboard points at exponential cost with a verifier-shaped ceiling. It’s an offline high-stakes batch technique, not a product feature, and if you can write the unit test, you don’t need the critic at all.
#MachineLearning #LLM #AIEngineering #MLOps #Inference #AIAgents #SoftwareEngineering
📚 Related Papers:
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Available at: https://arxiv.org/abs/2408.03314
- Let's Verify Step by Step?. https://arxiv.org/abs/2305.20050
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment. Available at: https://arxiv.org/abs/2503.21878
- Generative Language Modeling for Automated Theorem Proving. Available at: https://arxiv.org/abs/2009.03393