Machine Learning System Design Interview #16 - The P-Value Mirage

Why KS tests fail at scale and how magnitude-based drift metrics save your retraining pipeline.

You’re in a Machine Learning Research Engineer interview at Google DeepMind. The interviewer sets a trap:

“We need an automated trigger for model retraining based on feature drift. How do you implement it?”

90% of candidates walk right into the statistical trap.

In their mind, the textbook answer comes out automatically:

“I’ll run a 𝐓𝐰𝐨-𝐒𝐚𝐦𝐩𝐥𝐞 𝐊𝐨𝐥𝐦𝐨𝐠𝐨𝐫𝐨𝐯-𝐒𝐦𝐢𝐫𝐧𝐨𝐯 (𝐊𝐒) test between the training data and the live inference window. If the 𝘱-𝘷𝘢𝘭𝘶𝘦 drops below 0.05, the distributions are statistically significantly different. That triggers the retraining pipeline.”

It sounds rigorous. It feels scientific. It is also a guaranteed way to wake up your on-call team at 3 AM 𝘦𝘷𝘦𝘳𝘺 𝘴𝘪𝘯𝘨𝘭𝘦 𝘯𝘪𝘨𝘩𝘵.

The candidates are optimizing for 𝐒𝐭𝐚𝐭𝐢𝐬𝐭𝐢𝐜𝐚𝐥 𝐒𝐢𝐠𝐧𝐢𝐟𝐢𝐜𝐚𝐧𝐜𝐞, but in production, they must optimize for 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐒𝐢𝐠𝐧𝐢𝐟𝐢𝐜𝐚𝐧𝐜𝐞.