The Instructions, Not the Tools

The Trap of Better Tools

Napalys Klicius, the GitHub engineer who worked on this regression, described the trap perfectly: "Give an agent better tools and it should do better work. That's the instinct, anyway."

When GitHub decided to migrate Copilot code review's code exploration layer to the shared Unix-style tools (grep, glob, view) used by the Copilot CLI, the instinct was sound. Shared infrastructure means better maintenance, fewer duplicate implementations, and improvements that flow across products. The new tools were better tested, more general, and used by more teams.

The migration made things worse. Cost went up. Useful comments went down. The instinct was wrong.

Why Better Tools Made Things Worse

The shared tools were designed for a different job. The Copilot CLI handles open-ended interactive coding: understanding a repository, planning a change, editing files, answering questions over many turns. That context rewards broad exploration. An agent that searches widely, reads surrounding code, and accumulates context is doing exactly what it should.

Copilot code review has a narrower job: start from a pull request diff, gather just enough surrounding evidence to decide whether a change introduces a real issue, and stop. The agent should not browse the repository. It should interrogate the diff.

When GitHub dropped the shared tools in without adjusting the instructions, the agent kept behaving like a coding assistant. It searched broadly, guessed likely paths, read surrounding files, found more things to explore, and carried all that context forward. Every extra tool result became extra tokens in the context window. The review got more expensive and less focused.

The tools were not the problem. The instructions were giving the agent the wrong instincts.

The Fix: Review-Shaped Instructions

GitHub rewrote the tool instructions to match the review workflow. The change was small in wording and large in effect. The new instructions told the agent to:

• Start from the diff and ask targeted questions. What is the smallest code range that explains this behavior? What are the boundary conditions that matter?
• Use grep and glob to narrow the search, not to explore. Find the caller, find the usage, find the edge case.
• Batch focused reads instead of alternating between one search and one read.
• Recover from failed searches with a simpler corrected search, not by guessing neighboring paths.

The rhythm shifted from "browse, read, search again" to "ask, narrow, read, decide."

The result was an 11% reduction in tool calls and a 7% cost reduction, with no quality regression that could block shipping. The improvement did not come from the tools. It came from the workflow around them. The traces showed the agent spending more of its tool calls on relevant evidence instead of repeatedly expanding the search.

The Deeper Lesson: Tool Surface Is Product Surface

This is easy to miss when you are shipping fast. Tools look like implementation details. Swap one set for another, run the benchmark, compare the score. If the score does not move, call it done.

But for an agent, the tool surface is part of the product experience. It changes what the agent notices, how it searches, how much context it carries forward, and when it decides it has enough evidence. Tool descriptions and system instructions are closer to API documentation than to code libraries. Unclear API docs lead developers to make inefficient or wrong calls. Unclear tool instructions do the same for an LLM.

The GitHub team had the right read on the gap: "Shared tools scale when the instructions and benchmarks match the job."

When Shared Tools Work and When They Don't

GitHub tried applying the same focused tool instructions in the Copilot CLI and found they did not produce the same kind of win. That is a useful counterexample.

The CLI handles open-ended coding tasks where the user changes direction over multiple turns. There may be no single anchor point. The right context may not be obvious at the start. Broad exploration is part of the job. The same grep, glob, and view tools serve both products, but the workflow around them has to match the product.

This pattern generalizes beyond AI. Think about how your team uses the same monitoring stack for debugging production issues and for capacity planning. Same tools, different workflows. The teams that treat them as one thing get ambiguous dashboards. The teams that tune the workflow to the job get faster incident response and better planning data.

What Every Builder Should Steal

If you are building on top of an agent framework or shipping a product that wraps one, here is what this case study suggests:

1. Test tool instructions as aggressively as you test the tools themselves. A swap that looks clean in the diff can break the agent's behavior in invisible ways. The tool surface is part of the product; treat it like one.

1. Make agent traces visible. GitHub's internal benchmarks showed not just scores but the path the agent took: which tools it called, how much output came back, where errors happened, and whether it was narrowing or widening the search. You cannot fix what you cannot see.

1. Match the workflow to the task. The same tools that work for a coding assistant will produce different results in a code reviewer because the job is different. Tune the instructions for the job, not for the abstraction layer.

1. A regression after a tool swap is not a tool problem until you prove it is. Check the instructions first. The GitHub team wasted time debugging the tools before realizing the prompt was the culprit.

1. Build a benchmark that shows the path, not just the score. A final accuracy number hides agent behavior. GitHub's benchmark traced every tool call, every failure, every widening loop. That trace was the difference between guessing and knowing what was wrong.