Machine Learning System Design Interview #41 - The Average Feature Trap

The dangerous fallacy of using global metrics to justify individual edge cases, and how elite engineers build real-time local explanation pipelines to save high-value user experiences.

You’re in a Principal AI Engineer interview at Stripe and the interviewer asks:

“The business team refuses to deploy your new fraud detection neural network because they can’t explain to regulators why a specific high-value user was blocked. If a candidate hands them a global feature importance plot, why does that completely miss the mark, and how do you fix it for an audit?”

Most candidates say: “I’ll pull up the global feature importance chart from our training run. It proves that transaction volume and IP velocity are the top risk factors across our entire dataset, which justifies the model’s behavior.”

Wrong approach. That gets your model shelved and your interview ended.

The reality is: Global feature importance is an average, and averages lie in edge cases.

Using a global chart to explain an individual fraud flag is like using a country’s average weather map to explain why it’s raining inside your apartment. It is completely useless for localized debugging and regulatory compliance.

Regulators don’t care about your dataset’s average. They care about this specific user. In production, you need local interpretability, the exact micro-decision drivers for a single data point.

Here is how we should build an audit-ready explanation pipeline: