LLM Inference Interview Questions #6 - The Multimodal Perception Trap

Why dropping frontier VLMs onto dense web UIs is a hidden trap, and how foundation labs use cheap synthetic data to turn a prompting party trick into reliable execution.

You’re in a Senior AI Engineer interview at Google and the interviewer asks:

“Your web agent uses set-of-marks, you screenshot the page, draw numbered boxes on every element, and let the VLM click by number. It works in your demo but in production the model keeps clicking box 41 when it meant box 14, and it ignores half the page. The VLM is state-of-the-…