AWS Details Multi-Agent Testing Strategy to Stop Silent AI Failures

The framework tracks five operational dimensions: safety, quality, performance, cost, and operational control. To measure quality rigorously, AWS established a taxonomy of six specific failure categories:

• Metric swap: Citing an incorrect measurement despite getting the customer and timeframe right.
• Wrong period: Conflating intervals, such as reporting year-over-year numbers as month-over-month figures.
• Wrong entity: Confusing subsidiaries or similarly named corporate accounts.
• Pure hallucination: Introducing figures completely unsupported by underlying sources.
• Wrong symbols: Displaying inaccurate currency or measurement units.
• Unauthorized calculation: Generating derived statistics absent from source datasets.

Scoring combines deterministic checks (for verifiable ground truth like revenue or contract dates) with LLM-as-judge evaluations for subjective qualities like relevance, instruction adherence, and helpfulness, drawing on both native AgentCore templates and third-party tools like DeepEval and AutoEval.

The pre-production testing pipeline and measurable results

AWS designed a nine-stage evaluation lifecycle to catch degradation at every phase of code delivery. During continuous integration (CI), every deployment to staging automatically triggers a run against a focused "sentinel set" of 15 high-risk scenarios. This lightweight check blocks regressions instantly. For example, when a data normalization change caused an entity disambiguation tool to fail on certain account formats—leading the agent to guess customer names—the sentinel check failed and blocked deployment before the bug reached production.

Before releasing to production, continuous delivery (CD) gates execute deeper validation against a comprehensive golden dataset. In this stage, tool calls are sandboxed: the agent interacts with frozen, version-controlled tool outputs rather than live databases, isolating the agent's multi-step reasoning capabilities from external network latency or tool downtime. Candidates must also complete a holistic evaluation across 200 held-out questions scored by automated judges.

This structured feedback loop produced measurable improvements over a three-month optimization period. Overall accuracy pass rates rose from 93 percent to a consistent 96 percent. Simultaneously, pure hallucinations fell from 138 occurrences per 10,000 responses to under 52, while unauthorized calculations decreased from 59 to under 44 per 10,000.

What it means for developers

The AWS architecture demonstrates that standard DevOps monitoring—tracking error status codes, CPU load, and latency—is insufficient for autonomous agents. Because agent components degrade independently, teams must enforce evaluation contracts at the tool level, the sub-agent level, and the multi-agent coordination level.

Furthermore, system changes must be carefully synchronized. AWS observed that when data schemas, prompt updates, and tool logic were deployed out of sync in pre-production testing, the system pass rate plummeted from 78 percent down to 38 percent. Establishing versioned regression test sets and mock replay environments allows engineers to safely iterate on prompts and underlying models without introducing unpredictable side effects.

For engineering teams experimenting with multi-agent orchestration and balancing reasoning quality against token expenses, developers can try top AI models cheaply through one API at https://apixoai.online to benchmark latency, accuracy, and tool-calling capabilities across different model providers.

Source: Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore | Amazon Web Services — AWS Machine Learning. Written by the Apixo team from that report.