LLM System Design Interview #12 - The MoE Collapse Trap

When your Mixture of Experts stops learning, it's not the optimizer - it's the router. How expert starvation silently turns a 500B model into a 50B one (and how to fix it).

You’re in a Senior ML Engineer interview at Google DeepMind and the interviewer asks:

“You have just launched a new Mixture of Experts (MoE) training run. After a few thousand steps, you check the logs and see the validation loss has flatlined. What is the 𝐦𝐨𝐬𝐭 𝐥𝐢𝐤𝐞𝐥𝐲 𝐜𝐚𝐮𝐬𝐞 specific to an MoE, and how do you fix it?”