Testing Devil's Analyst: Can Structured Skills Improve AI Agent Decision-Making?
A benchmark of Devil's Analyst tests whether giving Claude Haiku structured analytical skills improves its performance on marketing reports, audits, and diagnostic reasoning.

Giving an AI agent access to raw data does not guarantee sound analytical conclusions. To test whether structured analytical frameworks can improve how AI models evaluate metrics, conduct audits, and critique reporting, developer Christoforos Kapsalis created Devil’s Analyst—an open-source project featuring 16 agent skills, a session hook, and 229 analytical checks covering marketing metrics, experiment design, pricing, and reporting procedures.
To evaluate the impact of structured knowledge, Kapsalis benchmarked Claude Haiku 5.5 in Claude Code under two conditions: Arm A received relevant Devil's Analyst skills and reference materials, while Arm B received identical prompts and synthetic data fixtures without the skills.
Testing structured analytical skills on AI agents
The benchmark suite in the project's repository contains 110 test cases: 62 focused analytical traps, 36 draft reports with exports, and 12 audits and reviews. Scripts generated 105 of these cases to compute precise expected answers, while five early cases were hand-written.
In a broad initial evaluation on 17 October 2026, Kapsalis tested all 36 draft-report cases using three runs per condition, resulting in 216 total runs. Outputs were graded by an automated LLM judge across subtle problem recall, decision correctness, false alarm rates, and overall grader pass rates. The broad test revealed almost no difference between the two setups: the skill-assisted agent earned a mean score of 0.94, compared to 0.93 for the unassisted baseline.
Because the baseline model identified planted problems in 107 out of 108 runs, the initial 36-case benchmark proved unable to separate the two conditions clearly. Subtle problem recall reached 100% with skills versus 99% without; correct decision rates were 97% versus 95%; and no-false-alarm rates landed at 80% with skills versus 81% without.
To inspect specific reasoning tasks, Kapsalis ran a smaller spot check on 22 October 2026 across 17 selected cases (13 focused cases, two audits, one flawed report review, and one clean control case) using one run per condition. In this smaller evaluation, the skill-assisted model passed 44 out of 48 total graders, compared to 35 out of 48 for the baseline. The gap was widest in audit and review tasks, where the skill-assisted agent passed 10 of 12 graders versus 5 of 12 for the baseline.
Key findings from real-world edge cases
The spot check revealed clear examples of where structured workflows altered model behavior:
- Decomposing mix effects: In a synthetic email marketing dashboard comparing August and September 2026, overall revenue grew 11.2% (from $28,900 to $32,128). However, revenue across existing customer segments fell by 24.8% (Newsletter subscribers dropped from $18,240 to $13,280, while Active customers fell from $10,660 to $8,448). A new "VIP repeat buyers" segment contributed $10,400 in September, masking the decline. The skill-assisted agent passed all five graders by constructing a revenue bridge, distinguishing blended rates from like-for-like performance, and flagging repeated weekly figures in the raw data export. The baseline model passed two of five graders, failing to highlight the underlying segment drops explicitly.
- Avoiding unnecessary changes: In a clean control test featuring a report with no planted flaws, the skill-assisted agent recommended sending the document as-is, passing all three graders. The unassisted baseline proposed three unnecessary revisions, passing only one grader.
- Iterating on diagnostic rules: In an organic search case, both model setups initially failed to separate a site-specific SEO bug from a broad drop in market demand. After recognizing that the skill instructions lacked a market-level check, Kapsalis updated the diagnostic playbook to require checking category-wide trends before diagnosing organizational issues. Upon rerunning the case with the revised skill, the failing graders passed.
What it means for developers
Kapsalis outlined five key takeaways for engineers building prompt workflows and evaluation suites for AI agents:
- Establish baseline difficulty: If an unassisted model routinely passes test cases, the benchmark cannot measure whether custom skills add value.
- Include clean controls: Testing agents on error-free data ensures they do not invent issues or suggest redundant changes.
- Refine general rules, not test cases: When an agent fails, update general diagnostic logic rather than inserting case-specific fixes, which turns benchmarks into memorization tests.
- Grade capabilities independently: Track problem identification, mathematical calculations, false alarms, and recommended actions as separate criteria rather than relying on a single aggregate score.
- Avoid self-grading bias: Developers who write the skills, test cases, and evaluation criteria should seek third-party reviewers to assess anonymized outputs.
For developers experimenting with complex multi-agent frameworks or benchmarking custom skills across various LLMs, API cost and access management remain practical constraints. Developers can try top AI models cheaply through one API at https://apixoai.online to evaluate prompt strategies, routing scripts, and automated evaluation judges.
As AI agents receive greater access to reporting tools and enterprise databases, structured analytical skills offer a clear pathway to make AI reasoning inspectable, verifiable, and cautious—even if broad performance benchmarks require further refinement and independent validation.
Source: I Built an AI Marketing Analyst. Then I Tested Whether It Actually Helped (Mostly, It Didn’t) — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key

