AI Interview Prep

AI Interview Prep

LLM System Design Interview #25 - The Leaderboard Illusion

Why chasing Chatbot Arena ELO degrades your model, burns compute, and misaligns you from real customers, and what a senior AI engineer should optimize instead.

Hao Hoang's avatar
Hao Hoang
Nov 23, 2025
∙ Paid

You’re in an AI Engineer interview at Anthropic and the interviewer asks:

“Our model is lagging on the Chatbot Arena leaderboard. We need to boost our ELO score by 50 points next release to match GPT-5. How do you adjust the post-training pipeline to make this happen?”

Don’t say: “We need to improve our reasoning capabilities on hard math benchmarks like …

User's avatar

Continue reading this post for free, courtesy of Hao Hoang.

Or purchase a paid subscription.
© 2026 Hao Hoang · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture