AI Coding Agents Fail to Complete 74% of Real-World Tasks, New Benchmark Shows
A new benchmark by researchers from Alibaba and Asian universities reveals AI coding models leave tasks incomplete 73.6% of the time, requiring manual developer intervention.
Existing coding benchmarks have increasingly shown AI models achieving near-perfect scores above 99 percent, but real-world performance often paints a very different picture. A new benchmark called Co-Design Bench, created by researchers from Alibaba and several Asian universities, demonstrates that even top-performing AI coding agents fail to deliver fully working code on nearly three out of four tasks. In real-world software maintenance scenarios, human developers are still required to manually clean up incomplete or flawed outputs in 73.6 percent of cases.
A benchmark built from real developer friction
The research team developed Co-Design Bench to address the saturation of older evaluation frameworks like WebCompass-edit, where self-checking models routinely hit maximum scores and obscure performance gaps exceeding 35 percentage points. Unlike synthetic or simplified tests, Co-Design Bench relies on actual change requests collected from real coding sessions. The dataset specifically prioritizes complex tasks where human users previously had to rephrase instructions multiple times or roll back broken edits.
These tasks range from minor user interface tweaks—such as pulling a dollar figure from a receipt to recalculate dependent totals—to full-scale repository overhauls. In multimodal scenarios, agents must parse information across tables, audio recordings, documents, or video files and update all dependent application views consistently. To prevent models from passing via superficial fixes, the benchmark employs a strict two-stage AI evaluation process. A primary AI judge executes and tests the final application inside a browser, while a secondary AI judge inspects the code to flag hardcoded values or non-functional mock-ups. Points are awarded only when features function correctly within the application environment.
Multi-faceted tasks expose model limitations
Across the ten models evaluated, GPT-6 Astra achieved the highest overall score. It fulfilled an average of roughly 70 percent of task requirements, yet it completely solved only 26.4 percent of tasks without errors. On specific subsets, GPT-6 Astra completed 41 percent of multimodal tasks and just under 18 percent of repository-level overhauls. Following GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max secured the second and third positions, while Grok 4.6 ranked last. The smaller Qwen3.8-27B model performed well on larger project overhauls, taking second place in that category, though the authors cautioned against drawing broad conclusions from that specific result.
The evaluation revealed a uniform drop in performance across all ten models whenever tasks combined three or four distinct change types, such as simultaneous edits to layout, visual design, user interaction, and underlying content. Because combined requirements represent standard practice in software engineering, this weakness highlights a major hurdle for current agentic workflows. Furthermore, allocating extra compute did not automatically resolve these shortcomings. For instance, GLM-5.2 consumed nearly six times as many tokens as GPT-6 Astra on smaller modifications while achieving significantly lower accuracy.
What it means for developers
The findings reinforce that high scores on public leaderboards do not translate directly to automated, hands-off software development. Real-world tasks frequently require human oversight to fix missing details, edge cases, and partial implementations. A separate study by research organization METR similarly highlighted this discrepancy, finding that human project leads accepted AI-generated code at rates roughly 24 percentage points lower than scores recorded on SWE-bench Verified.
The benchmark also highlighted the critical role of execution environments, or harnesses—the software tools that enable AI agents to read, write, and run code. When tested in different working environments, Qwen3.8-27B's success rate on large overhauls swung between 56 percent and 75 percent. Because model rankings shifted depending on the execution framework, the researchers recommend that teams evaluate AI models and their harnesses as a combined system rather than relying on model-only leaderboards.
As engineering teams evaluate how different models and execution environments perform on complex software tasks, cost-effective experimentation remains key. Developers can try top AI models cheaply through one API at https://apixoai.online to test how various providers manage real-world repository updates.
Despite its rigorous design, the authors note several limitations to Co-Design Bench. The test assesses single, fixed user requests without allowing follow-up prompts or mid-task corrections, which are common in actual development cycles. Additionally, AI judges are not entirely immune to grading errors, and prior research shows mixed evidence regarding the benefit of complex harnesses, with some setups consuming up to three times more tokens without yielding measurable improvements.
Source: AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test — The Decoder. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
