LLM Agents Interview Questions #16 - The Vision Encoder Scaling Trap
Fine-tuning the ViT won’t break the 20% ceiling because the bottleneck isn’t perception, it’s the missing symbolic bridge between visual claims and provable geometry.
You’re in a Senior AI Engineer interview at Google DeepMind. The interviewer sets a trap:
“You upgraded your geometry autoformalization pipeline from a 70B text-only LLM to a state-of-the-art VLM. You feed it textbook diagrams alongside the text. Success rates barely nudge past 20%. Why?”
90% of candidates walk right into it.
Most candidates say: “The vision encoder is losing spatial granularity. We need to unfreeze the ViT and fine-tune on higher-resolution diagram crops to capture exact intersection points.”
But they aren’t optimizing for pixel resolution, they’re optimizing for formal logic.
The reality is that human textbook diagrams contain unspoken assumptions. When a human looks at two overlapping circles, “intersection” is obvious. When a formal system like Lean evaluates a proof, “obvious” doesn’t compile. The VLM is accurately seeing the image, but it lacks the mathematical framework to formally prove what it sees.
