Advanced NLP Interview Questions #18 – The Perplexity Tokenizer Trap

Why lower perplexity across models can be pure metric illusion - and how tokenizer choice silently breaks probability comparisons.

You’re in a Senior AI Engineer interview at Google and the interviewer drops this on you:

“We ran an eval on a fixed dataset. Llama3 achieved a perplexity of 2.1, while Gemma3 scored 2.4. Which model is the better probability estimator, and which one do we deploy?”

Most candidates say:

“Llama3 is better. Perplexity measures how ‘confused’ the model is, so a lower score means it understands the text distribution better. We should deploy Llama3.”

They just fell for the classic 𝐓𝐨𝐤𝐞𝐧𝐢𝐳𝐞𝐫 𝐓𝐫𝐚𝐩. They treated a unit-dependent metric as a universal constant.

Here is the reality: Raw perplexity comparisons between different families of models are statistically meaningless.

Why? Because Perplexity is calculated per token, not per sentence.