Machine Learning System Design Interview #3 - The Gradient Drowning Trap

The hidden gradient dynamics that SMOTE, class weights, and oversampling can’t fix.

You are in a Senior Machine Learning interview at Google DeepMind. The interviewer sets a trap:

“We have a 1:1000 class imbalance for fraud detection. We applied 𝘤𝘭𝘢𝘴𝘴_𝘸𝘦𝘪𝘨𝘩𝘵𝘴 to the 𝐂𝐫𝐨𝐬𝐬-𝐄𝐧𝐭𝐫𝐨𝐩𝐲 loss, but the model is still missing the hard edge cases. What do we do?”

90% of candidates walk right into the wall.

Most candidates immediately suggest aggressive oversampling (𝘚𝘔𝘖𝘛𝘌) or tuning the class weights even higher (e.g., 1:5000).

They think: “If the minority class is ignored, I just need to scream louder (higher weights) during backprop.”

------

𝐓𝐡𝐞 𝐑𝐞𝐚𝐥𝐢𝐭𝐲:

You aren’t losing because the weights are wrong. You are losing because of 𝐆𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐃𝐫𝐨𝐰𝐧𝐢𝐧𝐠.