XYBench Benchmark Exposes LLM Weakness in Handling Misguided User Requests

To evaluate responses, XYBench relies on three core metrics grounded in pragmatics: pragmatic solution (whether a response includes a fix for Y, X, or both), pragmatic emphasis (which solution receives primary focus), and misconception identification (whether the model actively redirects the user away from the mistaken approach). Claude Sonnet 4.5 acts as an automated judge to grade responses. Human validation across 500 model responses confirmed high reliability for the automated judge, showing 97.5–100% agreement on solution presence, 84.6–93.3% on misconception identification, and 74.2–79.8% on pragmatic emphasis.

Key benchmark findings

The authors evaluated several leading models on XYBench in single-turn tests, including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro. Across all test sets, every model consistently prioritized addressing the user's literal request (Y) over their underlying objective (X).

On Stack Overflow questions, stated-request solutions appeared in 75% to 86% of model responses, whereas underlying-goal solutions appeared in only 59% to 71%. Furthermore, models focused primarily on the true underlying goal in just 35% to 41% of their responses, spending most of their output explaining the literal—but flawed—request.

The benchmark also revealed a distinct gap between model generation and model judgment. When the team presented models (including GPT-4.1) with a multiple-choice prompt comparing a literal answer against an answer addressing the misconception, models selected the goal-oriented answer 71% to 84% of the time. However, when generating responses from scratch, their success rate in emphasizing the underlying goal dropped significantly. Even providing explicit context about both X and Y in an oracle experiment yielded mixed results: while Claude Sonnet 4.6 increased its inclusion of underlying goal solutions from 59% to 67%, its rate of identifying misconceptions dropped from 47% to 34%.

What it means for developers

For software engineers building production-grade conversational agents, XYBench highlights a major blind spot in current LLM behavior. Standard benchmarks often reward models for answering queries literally, which can lead assistants to assist users down counterproductive paths.

The findings show that prompt engineering alone may not solve the problem. Supplying user intent explicitly helped models include correct solutions, but it did not consistently improve their ability to steer users away from bad approaches. Building assistants capable of active troubleshooting will likely require dedicated post-training strategies and fine-tuning targeted at intent inference and pragmatics.

Developers interested in testing how different frontier models handle complex troubleshooting queries can try top AI models cheaply through one API at https://apixoai.online. Evaluating models side-by-side on real-world prompts remains vital as teams build systems meant to handle open-ended user requests.

Moving forward, the benchmark creators note that expanding XYBench beyond English queries and procedural tasks will be crucial for broader AI safety and usability research. For now, XYBench offers a structured way to measure whether future model releases improve at guiding users toward real solutions.

Source: XYBench: Helping with the problem behind the question — Hugging Face blog. Written by the Apixo team from that report.