Machine Learning System Design Interview #11 - The ROC Curve Mirage

Why a “perfect” 0.98 ROC AUC can be worthless in fraud detection - and how PR AUC exposes the truth.

You’re in a Machine Learning interview at Google. The VP hands you a fraud detection model with a 0.98 ROC AUC and asks:

“Is this model ready to ship?”

90% of candidates walk right into the trap.

The textbook answer is to look at the score and celebrate. They said:

“0.98 is phenomenal. The curve hugs the top-left corner perfectly. The separation between classes is distinct. Let’s deploy.”