Machine Learning System Design Interview #20 - The Vanishing Update Paradox
Why increasing LoRA rank from 8 to 256 kills learning - and how rsLoRA fixes gradient collapse.
Youβre in a Senior ML Interview at OpenAI. The interviewer sets a trap:
βOur LoRA fine-tuning isnβt capturing the domain complexity. We increased the rank π« from 8 to 256 to give the model more capacity. But the loss curve flatlined. Why?β
90% of candidates walk right into it.
They say: βItβs overfitting. Rank 256 is too high for a ππ°πΈ-ππ’π―π¬ adaptation. The model is just memorizing noise, so we should reduce r back to 16 or 32.β
-----
ππ‘π ππππ₯π’ππ²: They arenβt overfitting. You are suffocating the model.
