Machine Learning System Design Interview #36 - The False Positive Blindspot

Why adjusting classification thresholds is just a superficial patch, and how to enforce a hard precision floor that survives real-world data distributions.

You’re in a Senior MLOps Engineer interview at OpenAI. The interviewer sets a trap:

“Your anomaly detection model boasts an incredible 0.98 ROC-AUC on an extreme 1:10,000 fraud-to-clean dataset. Yet, the moment it hits production, the team faces a massive flood of false positives. Why did your offline metric lie to you, and how do you fix it?”

95% of can…