LLM System Design Interview #9 - The Memory Wall

Why attention isn’t too slow - it’s too hungry. How FlashAttention wins by trading O(N²) memory I/O for O(N²) fast compute.

You’re in a Staff ML Engineer interview at a OpenAI and the interviewer asks:

“Your team needs to support a 10M context window. An engineer says it’s impossible because standard attention is O(N²) compute. Why is that the wrong bottleneck to focus on, and how does FlashAttention actually solve the real problem?”