Advanced NLP Interview Questions #11 – The Argmax Deadlock Trap

Hard argmax breaks differentiability in MoE routing, starving experts and freezing the gating network.

You’re in a Senoir Machine Learning interview at Google DeepMind. The VP of Engineering sets a trap:

“We are building a massive Mixture of Experts (MoE) model. To maximize training throughput on our H100 clusters, we want to route each token to only the single best expert (k=1). Is this a valid strategy?”

90% of candidates walk right into it. They immediately say:

“Yes. That’s the point of sparse MoE. If Expert A has a gating probability of 0.9 and Expert B is 0.05, calculating Expert B is a waste of FLOPs. Top-1 routing gives us maximum sparsity and speed.”

It sounds intuitive. It maximizes throughput.

It also guarantees their model will never converge.

The reality is they aren’t optimizing for inference speed yet; they are optimizing for differentiability.

The moment they select only the winner (k=1), they are performing a hard argmax operation. In the backward pass, argmax has zero gradient almost everywhere.

If a token goes to Expert A, Expert B receives zero signal. The router never learns “I should have sent this to B.” The unselected experts starve, the gating network freezes, and you end up with a massive, expensive model where only a fraction of the experts are actually trained.

-----

𝐓𝐡𝐞 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧: To pass the interview, you need to explain 𝐓𝐡𝐞 𝐀𝐫𝐠𝐦𝐚𝐱 𝐃𝐞𝐚𝐝𝐥𝐨𝐜𝐤.