LLM System Design Interview #22 - The Asynchronous Execution Trap
Why naive Python timers give you “1000× speedups,” why the GPU never actually ran your kernel, and the one missing line (synchronization) that every real ML engineer knows by heart.
You are in a technical interview at NVIDIA and the interviewer drops this scenario:
“An intern excitedly claims they achieved a 1000x speedup on a new matrix multiplication kernel. You look at their script and see they simply wrapped the function call with standard Python timers:
𝘴𝘵𝘢𝘳𝘵 = 𝘵𝘪𝘮𝘦.𝘵𝘪𝘮𝘦()
...
𝘦𝘯𝘥 = 𝘵𝘪𝘮𝘦.𝘵𝘪𝘮𝘦()
Why are their results a complete lie?”
Most candidates say...
“You need to use 𝘵𝘪𝘮𝘦.𝘱𝘦𝘳𝘧_𝘤𝘰𝘶𝘯𝘵𝘦𝘳() for better precision”
or
“They probably forgot to warm up the GPU cache before benchmarking.”
Wrong approach. That’s a generic optimization answer. You aren’t dealing with a precision error or a cache miss. You are dealing with a fundamental misunderstanding of how hardware talks to software.
The Reality: This is the 𝐀𝐬𝐲𝐧𝐜𝐡𝐫𝐨𝐧𝐨𝐮𝐬 𝐄𝐱𝐞𝐜𝐮𝐭𝐢𝐨𝐧 𝐓𝐫𝐚𝐩.
