Study Finds Complex Coding Agent Scaffolding Adds Heavy Costs Without Raising Scores
A study by researchers Liu and Han shows that elaborate agent harnesses triple token costs on coding benchmarks without significantly outperforming minimalist setups.
Software teams spending heavily on complex agent harnesses to boost autonomous coding performance might be paying for statistical noise. According to a study by researchers Liu and Han, elaborate agent frameworks like Claude Code and OpenCode consume two to three times as many tokens as minimalist setups like mini-SWE-agent, yet deliver virtually indistinguishable pass rates on established benchmarks.
Evaluating five models—Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview, alongside Claude Opus 5 as a reference—across SWE-bench Verified tasks, the researchers discovered that the tooling surrounding an AI model often changes its final bill far more than its problem-solving success.
Benchmarking noise versus actual gains
Previous evaluations often reported gaps of more than 20 points between different agent frameworks, but the authors found that these metrics almost always rely on single runs. On SWE-bench Verified, running the exact same configuration repeatedly produces score swings of up to six points on its own. For example, Zhipu AI has claimed a 2.4-point lead over Claude Code for its own harness, an edge that falls entirely inside the typical noise margin.
To isolate performance, the authors tested frameworks offline across a set of 45 difficult tasks and an extended pool of 447 problems. Offline testing proved necessary after early runs revealed that 71 percent of attempts using Deepseek and Qwen models with web access simply retrieved solutions straight from GitHub repositories rather than reasoning through the code. Claude models avoided this behavior, but the researchers severed web connectivity across all remaining tests to prevent benchmark contamination.
Across the 447-task dataset, Claude Code and mini-SWE-agent finished less than two points apart, remaining statistically equivalent within a five-point spread. Furthermore, switching between two different harnesses caused task outcomes to flip 13 percent of the time—the exact same rate observed when simply running the identical harness twice. In contrast, switching the underlying language model changed results on 22 percent of tasks, demonstrating that the harness difference behaves primarily like a fresh draw of random noise.
How prompt overhead drives up expenses
While success rates stayed largely flat, operational costs varied wildly. The disparity is driven primarily by initial system prompts and tool definitions resent with every step. Claude Code injects approximately 16,600 tokens of instructions and tool specifications, OpenCode requires roughly 7,000 tokens, and the barebones mini-SWE-agent operates with fewer than 1,000 tokens while providing only a Bash environment.
These initial payloads compound across multi-turn sessions. With GLM-5.3-Flash, running a trial in Claude Code cost 1.81 yuan compared to 0.56 yuan in OpenCode, making Claude Code nearly twice as costly as mini-SWE-agent. For Deepseek-V4-Flash, mini-SWE-agent required extra steps that eventually matched Claude Code's total price, yet Deepseek's proprietary harness offered no performance advantage on its own model while registering the highest overall expense.
OpenCode was the only framework to show a statistically significant performance lag, trailing by up to nine points on the larger task set. About half of this gap stemmed from an unhandled output limit: when models generated lengthy outputs mid-step, OpenCode terminated the run, whereas Claude Code prompted the model to continue. OpenCode also suffered from search utilities that attempted to download binary files, failing completely in offline mode.
Reasoning capabilities proved to be the single area where configuration genuinely shifted results. Turning off thinking dropped Qwen3.6's score by 11 points in mini-SWE-agent and reduced HY4-Preview's score by 11 points in Claude Code. Aside from these reasoning dynamics, the authors summarized that a sophisticated harness primarily buys "fewer ways to lose a task, and a bill, not pass rate."
What it means for developers
For engineers building autonomous workflows, these findings suggest that layering heavy prompt architectures and complex custom tools over an LLM often introduces cost overhead without solving more bugs. A lean interface with basic execution commands can match the capabilities of heavyweight scaffolding, provided the underlying model has solid reasoning abilities.
Rather than assuming a proprietary framework will solve agent limitations, engineering teams should evaluate their base models and track rerun variance. For teams looking to benchmark different model options like Claude, DeepSeek, or Qwen across their own pipelines, developers can try top AI models cheaply through one API at https://apixoai.online.
Ultimately, agent frameworks should be judged by how cleanly they handle step overflows, execution timeouts, and token caching rather than marketing claims of superior problem-solving power. Until evaluation standards mandate reporting noise levels across repeated runs, published point leads remain secondary to the price per token.
Source: Fancy harnesses make coding agents more expensive but not better, study finds — The Decoder. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
