Advanced NLP Interview Questions #3 – The Attention Entropy Illusion

Attention looks like a probability distribution, but it cannot tell you how certain the model is, here’s the hidden flaw.

You’re in a Senior ML Interview at OpenAI. The interviewer points to a Transformer diagram and sets the trap:

“How do we use the Attention mechanism’s weights to measure the model’s uncertainty?”

90% of candidates walk right into the trap.

“Easy. The Attention scores pass through a Softmax. They sum to 1.0. Therefore, they represent a probability distribution. If the attention is peaked on one token, the model is confident. If the distribution is flat (high entropy), the model is uncertain.”

This answer sounds intuitive. It is also mathematically invalid.

They just confused a 𝐌𝐢𝐱𝐢𝐧𝐠 𝐖𝐞𝐢𝐠𝐡𝐭 with a 𝐑𝐚𝐧𝐝𝐨𝐦 𝐕𝐚𝐫𝐢𝐚𝐛𝐥𝐞.