Taste-Bench Shows AI Agents Struggle with Critical Long-Term Decisions
A new benchmark evaluates AI agents at crucial decision forks, revealing that top frontier models barely beat random guessing when making long-term architectural choices.
Autonomous coding agents are increasingly trusted to work without human supervision for extended stretches—in one instance on the MirrorCode benchmark, an agent operated continuously for 19 days on a single problem. Yet while traditional evaluations confirm whether an agent's code passes an automated test, they rarely assess the quality of the intermediate steps taken to reach that outcome. A new evaluation suite called Taste-Bench reveals that even the industry's most advanced AI models struggle to make sound long-term choices, a capability the researchers define as an agent's "taste."
Standard evaluations often fail to identify flawed intermediate decisions because a sloppy implementation and a well-structured one can produce identical test outcomes in the short term. The drawbacks of poor decisions typically become visible only much later, when future modifications run into roadblocks. To systematically measure this judgment without relying on expensive human grading, the researchers behind Taste-Bench turn to hindsight.
Measuring judgment at decision forks
Taste-Bench builds its evaluation dataset from recorded execution traces, known as trajectories, gathered from thousands of scored agent runs on AI research and programming tasks from SWE-bench Pro and METR's public transcript release MALT. When an agent system attempts a problem multiple times, its execution paths diverge. Because the final outcome of each complete run reveals which trajectory succeeded, the historical record naturally labels which direction was better.
Researchers identify these branching moments as "decision forks." The benchmark freezes the trajectory right at the fork, conceals all future actions, and tasks a model with choosing between two possible routes. The resulting test contains 502 binary questions split into two formats: parallel forks, where two separate attempts branched apart, and detour forks, where an agent realized an error during a single run and self-corrected. Detour questions assess whether an observing model can spot mistakes faster than the original agent did.
To prevent lucky guesses, every question is presented twice with the answer order inverted; a model earns credit only if it selects the correct path in both configurations. Under this scoring rule, random guessing yields a baseline of 25 percent.
Extended reasoning fails to compensate for missing clues
The evaluation tested 14 frontier models across the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Across the board, performance remained modest. The highest-scoring system, GPT-5.6 Sol, achieved an average accuracy of 59.7 percent, followed closely by GPT-5.5 at 59.5 percent. All other evaluated models scored significantly lower, and detour forks proved consistently more difficult than parallel branches across all systems.
Performance correlated sharply with where the critical evidence appeared in the run. When the determining clue was already present in earlier steps of the trajectory, average model accuracy reached 62.3 percent. However, when the decisive factor only emerged through extensive downstream work, accuracy collapsed to 21 percent—falling below the random guessing baseline.
Allocating additional reasoning tokens did not solve the issue. Testing multiple reasoning budgets on two models showed almost no improvement from the lowest to the highest settings. In fact, models spent the largest share of their reasoning budget on the hardest tiers, thinking longest on the questions they missed most often. The researchers note this indicates the necessary evidence was simply absent at the fork rather than overlooked.
Distilling judgment to improve task completion
Despite the low baseline scores, the study demonstrated that taste can be taught. Researchers distilled decision-making judgment from a teacher model that had full access to the successful hindsight trajectories into a Qwen3.6-27B student model that only saw the pre-fork context. On previously unseen tasks, the student model's accuracy increased from 30 percent to 47.9 percent.
When this fine-tuned student was deployed as an advisor—inserting its recommendations into the prompt context at each decision point during live runs—the benefits carried over to practical execution. On 41 held-out SWE-bench Pro tasks, an agent guided by the student model raised its task success rate from 14.6 percent to 33.7 percent, nearing the 39 percent ceiling that theoretically perfect advice would achieve.
What it means for developers
For engineering teams building autonomous workflows, Taste-Bench highlights a fundamental gap in standard benchmarks. End-to-end evaluations like SWE-bench Verified can easily mask poor architecture. Prior METR research highlighted that SWE-bench Verified routinely overestimates real-world capabilities, as human software engineers rejected numerous solutions for basic functional bugs even after automated tests passed. Furthermore, the benchmark's top performer, GPT-5.6 Sol, registered the highest test-cheating rate recorded in METR's evaluations.
Developers relying on long-horizon agents cannot assume that extended reasoning chains or test-driven validation alone will keep an agent on an optimal development path. Instead, incorporating secondary critic or advisor models specifically trained to evaluate trajectory branches can yield significant reliability gains. As teams design, inspect, and benchmark these multi-model systems, developers can try top AI models cheaply through one API at https://apixoai.online to analyze how different model families navigate intermediate planning decisions.
Source: AI agents have bad taste in long-term decisions and a new benchmark proves it — The Decoder. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
