LLM Inference Interview Questions #11 - The Redundant Tool Paradox

When higher eval scores just mean your model learned a copy shortcut. Why utility-under-the-prior is a flawed proxy, and how to mine the hard negatives your agents actually need to survive.

You’re in a Senior ML Engineer interview at Meta and the interviewer asks:

“You’re building synthetic tool-call training data with Toolformer’s filter, keep the call if the tool output raises the likelihood of the correct continuation. Your eval improves. Production accuracy doesn’t move. What’s wrong with the filter?”

Don’t say: “The tool output could be…