LLM System Design Interview #1 - The Tokenizer Trap

Why reusing a general tokenizer can quietly 16x your compute bill - and how to avoid it.

You’re in an AI Engineer interview at Anthropic and the interviewer asks:

“We’re training a new model for the legal and medical domain. What’s the production risk of just using a standard, pre-trained Llama 3 tokenizer, and what’s your fix?”

Most candidates say: “It’s not optimal, but the tokenizer will just use subwords for the OOV (out-of-vocabulary) terms like ‘aneurysm’. The model will eventually learn the combinations during fine-tuning.”

Wrong. This answer ignores the real production bottleneck: compute cost.

The reality is that tokenization is a 𝐥𝐞𝐚𝐤𝐲 𝐚𝐛𝐬𝐭𝐫𝐚𝐜𝐭𝐢𝐨𝐧.