Why Minor Agent Benchmark Swings Are Often Statistical Noise
A deep dive into agent benchmarks reveals why small score shifts are often statistical luck, showing how paired noise bands and task categorization expose true model performance.
Engineering teams building AI agents frequently run into a familiar cycle: a pull request updates a system prompt, tweaks tool definitions, or wraps an existing model in a new CLI. The author notes a modest improvement in benchmark results, yet a week later, another team member observes a small dip and reverts the work. This churn often stems from ignoring a fundamental evaluation question: is a minor score change larger than the random variance the benchmark produces on its own?
An analysis by Paolo Perrone titled "How to Build an Agent Eval You Can Trust" (published October 31, 2026, in Data Science Collective) illustrates this challenge using public data from the Terminal-Bench 2.0 leaderboard on Hugging Face. Examining 89 tasks with at least five runs per task, Perrone recomputed the statistics to evaluate how much noise exists in standard agent evaluation setups.
Harness Changes Versus Real Model Drops
When evaluating Claude Opus 4.6 while switching harnesses from Terminus 2 to Copilot CLI, the pass rate moved from 62.9% to 60.4%—a drop of 2.5 points. However, the calculated noise band for that comparison was plus or minus 6.7 points. Because the 2.5-point shift fell well within this boundary, the benchmark could not mathematically distinguish the change from pure chance.
In contrast, swapping models under the same Terminus 2 harness showed a distinct pattern. Switching from Claude Opus 4.6 to Kimi k2.5 resulted in a 20.4-point drop against a noise band of plus or minus 7.8 points. Because the decline was nearly three bands wide, it represented an actual performance drop rather than statistical noise.
Even without changing any code or configurations, natural variance can be deceptively high. Across five full runs using Factory Droid with Claude Opus 4.6 on the same suite, the score showed a 3.4-point spread between its best and worst run. When an identical setup wobbles by more than 3 points against itself, taking a single run from each side of an A/B test cannot provide reliable conclusions.
Calculating Paired Noise Bands
To separate true performance changes from run variation, Perrone outlined a statistical approach based on paired task differences:
- Multiple runs per task: Run every task at least five times to establish a discrete pass rate (such as 0%, 20%, 40%, 60%, 80%, or 100%).
- Pair by task: Subtract the baseline pass rate from the candidate pass rate for each individual task.
- Mean change: Compute the average across all per-task differences.
- Standard deviation: Calculate how widely those task-level differences scatter.
- Sample size scaling: Divide the standard deviation by the square root of the total task count.
- Confidence threshold: Multiply the result by 1.96 for a roughly 95% confidence band.
Under this formula—band = 1.96 * stddev(per-task differences) / sqrt(number of tasks)—the plus or minus 6.7-point band over 89 tasks indicates an underlying per-task standard deviation of approximately 32 points. This indicates that while individual tasks swing by as much as a third of the scale in either direction between harnesses, those shifts largely cancel out in the aggregate.
Pairing tasks directly is critical because it controls for baseline difficulty. Comparing whole suites as unrelated pools would needlessly widen the noise floor. Furthermore, Perrone distinguished between two types of luck: run luck (flaky behavior on a single task across executions) and task luck (how well the chosen benchmark tasks represent the broader problem space). In the harness comparison, run luck accounted for only about a quarter of the variance. Even with infinite reruns, the noise band would have bottomed out near plus or minus 5.7 points. Shrinking the band down to plus or minus 2.5 points would require scaling the benchmark to roughly 650 tasks.
Task Classification: Solid, Flaky, and Broken
Looking strictly at average numbers can mask critical failures. Perrone suggested classifying tasks into three distinct populations based on their five-run outcomes:
- Solid: Passed all 5 runs.
- Flaky: Passed some runs, but failed others.
- Broken: Failed all 5 runs.
For Claude Opus 4.6 running on Terminus 2 across the 89 tasks, the breakdown showed 39 solid tasks, 30 flaky tasks, and 20 broken tasks. Although the aggregate pass rate was 62.9%, only 43.8% of tasks passed consistently on every attempt.
This distribution reveals why score improvements can be deceptive. A harness update might appear to boost scores simply by catching a few flaky tasks on a favorable run. Conversely, aggregate metrics can hide regression anomalies. In the Copilot CLI comparison, two specific tasks (circuit-fibsqrt and schemelike-metacircular-eval) lost all five of their previous passes. Spotting these shifts requires analyzing individual task traces rather than relying solely on high-level averages.
What it means for developers
For engineering teams building autonomous agents or automated testing pipelines, evaluating model or prompt updates requires the same rigor used in standard performance benchmarking:
- Stop relying on single-run evals: A single benchmark pass cannot differentiate between prompt optimizations and random chance.
- Implement CI noise gates: Tools like the standalone
NoiseBand.csscript can ingest run-by-run CSV logs, eliminate unpaired or under-run data, and automatically fail pull requests that claim improvements within the noise band. - Look beyond the mean: Track tasks that flip entirely from solid to broken to spot targeted functional regressions.
- Evaluate across model providers: If you are testing performance shifts across diverse architectures, developers can try top AI models cheaply through one API at https://apixoai.online, simplifying comparative testing across various model families without managing separate integrations.
Treating agent benchmark numbers as statistical distributions rather than static scores prevents teams from mistaking random run luck for genuine engineering progress.
Source: Your Harness Didn’t Get Better, It Got Lucky: Noise Bands for Agent Evals — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
