Machine Learning System Design Interview #21 - The Silent Feature Death
Why a perfectly healthy model can produce garbage A/B results when your feature freshness quietly collapses.
You’re in a Senior ML System Desgin interview at Facebook. The interviewer sets a trap:
“Your new pricing model has 99.9% availability, 15ms latency, and zero exceptions. Yet, the A/B test results are complete garbage. What happened?”
90% of candidates walk right into the wrong answer.
Most candidates immediately dive into the model weights or the experiment setup.
- “Is the model overfitting?”
- “Did we mess up the bucket randomization?”
- “Is there a logic bug in the inference code?”
They waste the entire interview debugging the model architecture. They assume that if the dashboard is green (HTTP 200 OK), the system is healthy.
-----
𝐓𝐡𝐞 𝐓𝐮𝐫𝐧: They aren’t optimizing for 𝘤𝘰𝘥𝘦 𝘤𝘰𝘳𝘳𝘦𝘤𝘵𝘯𝘦𝘴𝘴, they are optimizing for 𝘴𝘺𝘴𝘵𝘦𝘮 𝘴𝘵𝘢𝘵𝘦.
