Testing an AI Critic on Policy Claims and Complex Decisions
An exploration into using Jev, an AI model that cannot plan or write, as a policy critic inspired by Karl Popper's philosophy on error and open societies.
Philosopher Karl Popper argued that political questions should focus on limiting the damage caused by bad rulers through criticism rather than seeking ideal leaders. Applying this logic to modern technology, an experiment tested whether a specialized machine critic—a non-writing model named Jev from TypeSafe—could evaluate policy claims and detect reasoning errors in complex simulated scenarios.
Unlike traditional generative AI tools that draft, plan, or provide fluent explanations with hidden assumptions, Jev is designed solely to output calibrated probabilities based on specific criteria. Without the ability to generate prose or propose solutions, it functions strictly as an analytical checker. To see how such a critic performs, researchers subjected Jev to complex situational test cases inspired by psychologist Dietrich Dörner's studies on human failure in managing simulated towns.
Testing the Critic Against Complex Failures
Using 44 test cases derived from Harald Schaub's catalog of human thinking errors in complex situations, Jev was evaluated on criteria such as complexity, interconnectedness, dynamics, intransparency, multiple goals, and novelty. The model successfully handled 42 of the 44 cases, accurately navigating hidden variables and multi-step cause-and-effect chains.
However, it struggled with raw weekly numbers showing exponential growth and tended to falsely identify contradictions when analyzing extensive goal lists. By establishing a confidence threshold of 0.7—similar to C. K. Chow's 1970 classification rule—low-confidence errors could be automatically filtered and handed over to human supervisors, preventing blind acceptance of flawed automated outputs.
What it means for developers
Developers building evaluation pipelines or decision-support systems face the challenge of reliable verification. Standard language models often assert judgments with high fluency but lack calibrated probabilities, making them prone to unverified errors. Models like Jev demonstrate the utility of decoupling judgment from generation. By returning calibrated probabilities instead of unstructured prose, systems can programmatically filter, log, and route uncertain cases. Developers looking to experiment with specialized evaluation tasks and top AI models can try them cheaply through one API at https://apixoai.online.
Evaluating Policy Reasons and Real-World Friction
When applied to evaluate written policy justifications using Popper’s framework of testability, Jev successfully flagged unscientific immunization moves and untestable statements. However, when paired with an AI mayor (DeepSeek's flash model) governing a simulated town, coaching the AI to recognize logical flaws improved the quality of its written reasons without changing its actual policy decisions. The AI mayor could articulate refined justifications for its actions while repeating the exact same choices, illustrating that improving a policy's rhetoric is not the same as refuting the policy itself.
Ultimately, the experiment demonstrates that while an AI critic can assess whether a claim is testable or logically sound, only the real world—and structured code comparing expected results with actual outcomes—can determine whether a policy actually works.
Source: The Critic Who Cannot Plan — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
