Advanced NLP Interview Questions #10 – The Contrastive Batch Size Trap

In CLIP, small batches don’t just slow training, they mathematically break the loss.

You’re in a Machine Learning System Design interview at OpenAI. The interviewer sets a trap:

“We need to train a specialized CLIP model for medical imaging from scratch. You have a node of 8 A100s. What batch size do you configure?”

95% of candidates walk right into the trap.

The candidates pull out a calculator and optimizes for VRAM.

“Well, with a ViT-L backbone and high-res X-rays, a batch size of 64 per GPU is the safe limit to avoid OOM errors. So, a global batch size of 512.”

It sounds like a competent engineering answer. They respected the hardware constraints. They avoided the crash.

They also just guaranteed the model will be useless.

In Supervised Learning (e.g., ResNet classifier), batch size is just a gradient estimation tool. Smaller batches are often fine (or even better for regularization).

But in 𝐂𝐨𝐧𝐭𝐫𝐚𝐬𝐭𝐢𝐯𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 (𝐂𝐋𝐈𝐏), your batch size is your dataset.

- With a batch of N, you have N correct pairs (positives).

- But more importantly, you have N^2 - N incorrect pairs (negatives).