Advanced Deep Learning Interview Questions #4 - The I/O Starvation Trap

Scaling compute exposes that your pipeline is gated by data throughput, not model execution.

You’re in a Senior ML Engineer interview at Meta and the interviewer asks:

“You just migrated your team’s deep learning workloads from local hardware to a massive AWS GPU cluster to accelerate training. The expensive instances are successfully spinning, but your training iteration speed has actually flatlined. What is the hidden system bottleneck throttling your pipeline?”

Don’t say: “It’s a network latency issue. We just need to pay for a higher-bandwidth VPC or upgrade to faster compute instances.”

Wrong approach. You’re just throwing more cloud budget at the wrong problem.

The reality is that scaling up cloud compute almost always exposes the severe I/O Starvation of your data pipeline. You’ve essentially bought a fleet of Ferraris, but you’re trying to fuel them through a garden hose.