Advanced NLP Interview Questions #19 – The QLoRA Compute Tax Trap

Why 4-bit quantization saves VRAM but can slow down training on A100s and H100s.

You’re in a Senior AI Engineer interview at NVIDIA and the interviewer asks:

“We switched from standard FP16 fine-tuning to QLoRA (4-bit quantization) to save memory. The model fits now, but training speed hasn’t improved, it’s actually slightly slower. Why didn’t reducing precision by 4x result in a 4x speedup?”

Most candidates say: “That sounds like a bug. Since 4-bit integers are 4x smaller than 16-bit floats, we are moving less data. Less memory bandwidth usage always equals faster training.”

𝐖𝐡𝐲 𝐭𝐡𝐢𝐬 𝐟𝐚𝐢𝐥𝐬: They are confusing storage efficiency with computational throughput. They are assuming the GPU is doing math in 4-bit. It isn’t.

Here is the architectural reality: QLoRA is a memory optimization, not a compute optimization.

To understand why your training slowed down, you have to look at what is happening inside the GPU kernels.

1️⃣ 𝐓𝐡𝐞 “𝐔𝐧𝐳𝐢𝐩𝐩𝐢𝐧𝐠” 𝐓𝐚𝐱:

The weights are stored in VRAM as 4-bit integers (saving huge amounts of space). However, NVIDIA Tensor Cores generally do not perform training operations (backpropagation) directly in INT4.

Before every Matrix Multiplication (GEMM), the system must dequantize those 4-bit weights back into BF16 or FP16 in the GPU’s cache.