Advanced NLP Interview Questions #1 - The Learning Rate Warm-Up Trap

The hidden physics of why Transformers can't handle a 1e-3 LR at initialization.

You’re in a Senior ML Interview at Google DeepMind. The interviewer hands you a marker and sets the trap:

“We are training a 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳 from scratch using 𝘈𝘥𝘢𝘮. We set a constant Learning Rate of 1e-3. Predict the first 1000 steps.”

90% of candidates walk right into the trap.

Most candidates say: “It converges. 𝘈𝘥𝘢𝘮 is an adaptive optimizer, it adjusts per-parameter learning rates automatically. 1e-3 is a standard default. It might be noisy at first, but the loss will go down.”

The reality? Your loss curve doesn’t just oscillate, it explodes. You hit NaNs within 50 steps. You just wasted a cluster run.

Here is the physics of why: 𝘛𝘳𝘢𝘯𝘴𝘧𝘰𝘳𝘮𝘦𝘳𝘴, unlike 𝘙𝘦𝘴𝘕𝘦𝘵𝘴, lack strong inductive biases at initialization.

---