Advanced NLP Interview Questions #5 - The Speculative Decoding Illusion

You think batch=128 makes speculative decoding useless. The reality? Your model is secretly dying of memory starvation.

You’re in a Machine Learning Engineer interview at OpenAI. The interviewer sets a trap:

“We need to optimize inference for batch size 128. Should we use Speculative Decoding?”

90% of candidates walk right into the trap.

They answer: “No. At batch size 128, the GPU is fully saturated (compute-bound). Running a draft model just adds overhead and kills throughput.”

It sounds logical. It is standard textbook advice.

It is also completely wrong for modern workloads.

Here is the blind spot.

The candidates are assuming standard RAG (2k tokens). But in Long Context Inference (100k+ tokens), the bottleneck shifts violently.

The candidates aren’t stalled by matrix multiplication, they are choking on memory bandwidth.

-----

𝐓𝐡𝐞 𝐑𝐞𝐚𝐥𝐢𝐭𝐲:

- Model Weights are shared across the batch (loaded once).

- KV Cache is unique to every single request (loaded 128 times).