Machine Learning System Design Interview #24 - The Silent Graveyard Effect

Why 5 years of “big data” at Walmart hides the customers your model most needs to learn from.

You’re in a final round interview for a Machine Learning Engineer role at Walmart. The interviewer sets a trap:

“We have 5 petabytes of transaction history spanning 5 years. Train a model to predict next month’s purchases.”

90% of candidates walk right into the trap.

They say : “Awesome. More data equals better generalization. I’ll ingest the whole 5-year history, feature engineer Recency, Frequency, and Monetary value (RFM), and train a massive XGBoost model.”

The interviewer stops writing. They just failed.