LLM Inference Interview Questions #16 - The Warm-Start Trap

Why initializing your agent with SFT before RL guarantees a flatlined reward curve, and why skipping straight to cold-start RL builds a more resilient tool user.

You’re in a Senior ML Engineer interview at Anthropic and the interviewer asks:

“You’re training a tool-use agent. Do you SFT on expert tool-call traces before RL, or go straight to RL from the base model?”

Don’t say: “Always SFT first, a warm start gives the model a good initialization and RL just refines it.”

That’s the textbook answer. It’s also why your RL run is going to flatline at step 200.

Here’s what’s actually happening.

SFT on tool-call traces isn’t teaching your model when to use a tool. It’s teaching it what a correct-looking trace shaped like an expert’s looks like. Those are different skills, and only one of them survives contact with production.

Three things break:

• Entropy collapse. SFT sharpens the output distribution hard. Post-SFT, your model generates near-identical rollouts for the same prompt. Now run GRPO: you sample 8 completions, all 8 get the same reward, the advantage is zero, and your gradient is zero. You paid for exploration and got a deterministic parrot. RL can only reward behavior it manages to sample. If SFT already pruned the branch, RL will never find it.

• Off-policy distribution shift. Expert traces are clean. The tool always returns. The parameters are always right. Your model learns the happy path and nothing else, so when a file-edit tool fails on a malformed arg in prod, it has never seen recovery, only success. It retries the same broken call eleven times because that’s the only shape it knows.

• Behavioral cloning debt. You’re not initializing the policy. You’re constraining it. Cold-start RL lets the model discover a tool-use strategy that fits its own capabilities. SFT forces it into a strategy that fit the annotator’s.

The nuance that separates you from every other candidate:

Cold start isn’t universally better. It works when the base model can already emit syntactically valid tool calls often enough for the reward signal to land. If your base model can’t produce parseable JSON schema calls, RL gets zero reward on every rollout and never bootstraps, then you need SFT, or a format reward to carry the early phase before the correctness reward takes over.

The real question isn’t “warm or cold.” It’s: does my base model have enough latent tool-calling capability that exploration is cheaper than imitation?

---

The answer that gets you hired:

SFT buys format compliance and pays for it in exploration diversity. If your base model already clears the syntax bar, that trade is a bad one, cold-start RL with a format reward scaffold beats a warm start that’s already collapsed the policy you were trying to learn.

---

#MachineLearning #LLM #AIEngineering #ReinforcementLearning #MLOps #AIAgents #DeepLearning

---