Architecting AI Agents: How to Control Tool Selection and Avoid Framework Defaults
In an eight-part series on agent design, engineering the harness before tool execution is critical to keep models from selecting the wrong tools or failing silently.
Building reliable autonomous systems requires strict engineering around the model rather than relying solely on the intelligence of the core foundation model. In the second entry of an eight-part series analyzing agent design, the architectural focus turns to tool provisioning: specifically, how developers determine which tools an artificial intelligence model receives and how it locates the rest.
The comprehensive series divides agent architecture into two distinct domains: six sections examine the code wrapping the model, while two sections examine team practices outside the codebase. Under this framework, an agent is defined as a model situated inside a harness. The harness consists of the code surrounding the model, which dictates what the system reads back following each tool call and governs when an execution run ends.
The Architectural Risk of Framework Defaults
A critical takeaway from the initial installments of the series is the hidden risk of unmanaged harness code. When an engineering team does not explicitly build this harness layer, the underlying framework's defaults make those operational choices instead. As a result, upgrading a third-party framework can silently alter execution behaviors without the development team's knowledge or consent.
The first part of the guide focused on execution loops and failure recovery, analyzing how models process feedback when a tool call fails. It introduced a design review methodology applicable to any model or framework, pairing primary patterns with corresponding anti-patterns.
The second part targets the critical stage preceding execution: deciding which tools are supplied to the model. An AI model can only invoke a tool that is present in its request payload. When choosing a tool from that payload, the model relies on three key attributes: the tool's name, its description, and its specified arguments.
Four Methods for Populating Tool Requests
According to the analysis, exactly four entities or mechanisms can dictate which tools are placed into the model's request:
- The Builder: The developer explicitly chooses and sets the tools when constructing the agent.
- Pre-Call Code: Custom code executes before each invocation, evaluating system state to determine what goes into the payload.
- Model Tool Search: The model itself identifies and selects relevant tools through an integrated tool search process.
- External Servers: Connected servers dynamically introduce or remove tools while the agent is running.
If developers do not deliberately architect how these four inputs function, default framework configurations take over. Leaving these settings unmanaged increases the risk that an agent will trigger the wrong tool or make faulty runtime decisions.
What it means for developers
For engineering teams building production agents, the configuration of the model harness must be treated as a primary architectural concern. Developers should avoid leaving tool discovery and payload composition to implicit framework defaults, which expose systems to silent behavioral drift across framework updates.
Because the model's tool selection depends entirely on names, descriptions, and argument structures, developers must write precise tool schemas. Ensuring the model accurately identifies tools requires deliberate choices regarding whether tools are passed statically at build time, dynamically injected via pre-call logic, queried by the model, or adjusted at runtime by servers.
Evaluating how different foundation models respond to dynamic tool lists and search patterns is an essential testing step. Developers can try top AI models cheaply through one API at https://apixoai.online to benchmark how various model families interpret tool descriptions, arguments, and dynamic payload changes within their custom agent harness.
Source: Make Sure Your AI Agent Can Find the Right Tool — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
