Advanced NLP Interview Questions #17 – The Sparse Gradient Trap

Why SGD silently ignores rare tokens - and how Adam rescues them with variance normalization.

You’re in a Senior ML Engineer interview at OpenAI and the interviewer asks:

“We’re training a model on a massive vocabulary. Some critical domain terms appear only once every 10,000 documents. Why will standard SGD fail to learn weights for these rare features, and how does Adam specifically fix this?”

Most candidates say: “Adam is better because it uses momentum to converge faster.”

Too vague. They just described 90% of optimizers. They missed the core problem: 𝐒𝐩𝐚𝐫𝐬𝐢𝐭𝐲.

The reality is that SGD is 𝐟𝐫𝐞𝐪𝐮𝐞𝐧𝐜𝐲-𝐛𝐢𝐚𝐬𝐞𝐝.

In standard SGD, the parameter update is directly proportional to the gradient.

- 𝘙𝘢𝘳𝘦 𝘍𝘦𝘢𝘵𝘶𝘳𝘦 = 𝘙𝘢𝘳𝘦 𝘯𝘰𝘯-𝘻𝘦𝘳𝘰 𝘨𝘳𝘢𝘥𝘪𝘦𝘯𝘵.

- 𝘙𝘢𝘳𝘦 𝘎𝘳𝘢𝘥𝘪𝘦𝘯𝘵 = 𝘛𝘪𝘯𝘺, 𝘪𝘯𝘧𝘳𝘦𝘲𝘶𝘦𝘯𝘵 𝘶𝘱𝘥𝘢𝘵𝘦𝘴.

By the time the model has converged on frequent words (like “the”, “is”, “user”), the weights for your rare terms (like “heteroscedasticity”) are still basically random initialization. They were drowned out.