Why System Design Beats Prompt Engineering for Reliable AI Workflows

First, creative judgment must be separated from strict rule enforcement. Large language models excel at tasks requiring language nuance, tone selection, and editorial judgment, such as drafting product copy or humanizing commentary. Conversely, they struggle with strict mechanical rules like exact character counting. In a data reporting context, for example, expecting an AI to clean raw data, calculate week-over-week percentages, format tables, and draft executive commentary in one pass often leads to inaccurate math or missing columns. A better approach assigns mathematical and structural operations to standard code, leaving the model responsible only for qualitative narrative generation.

Second, fixed rules should be handled programmatically. Constraints regarding character limits, date formatting, rounding, and field requirements belong in standard code routines. In Jiang’s weekly summary example, numerical calculations and table generation were handed off to tools like pandas, allowing the model to focus strictly on explaining the final metrics in plain English. Reducing the model's scope leads to shorter prompts, decreased token consumption, and fewer points of failure.

Third, automated verification checks must be embedded directly into the execution pipeline. Rather than assuming model outputs are flawless because they read convincingly, developers should treat AI outputs like software deliverables that require validation. Jiang implemented post-execution validation rules to check listing lengths, required fields, and formatting. For analytical reports, verification checks confirm whether reported numbers align with table data, verify row counts, and ensure that directional commentary matches the numerical signs of reported changes.

Structural benefits over prompt bloat

Shifting from prompt engineering to system architecture yielded immediate practical improvements in Jiang's workflows. Output consistency increased, and token waste caused by repeated retries was reduced significantly. Furthermore, system maintenance became far more straightforward. When an output fails validation, developers can isolate the broken component directly rather than guessing how to alter an over-engineered prompt without breaking existing behavior.

While architectural checks do not eliminate occasional qualitative flaws—such as a poorly written title—they prevent simple language errors from compounding into structural failures like broken limits or missing fields. For engineering teams optimizing these multi-step pipelines, managing token expenses across various foundation models remains a priority. Developers can try top AI models cheaply through one API at https://apixoai.online, which simplifies testing structured workflows across different LLMs.

What it means for developers

Jiang’s findings suggest that developers should reframe how they address unreliable AI behavior. Before attempting to fix an erratic workflow by expanding a prompt, engineers should ask three diagnostic questions:

1. Which components require linguistic judgment, and which are strictly rule-based?
2. Can traditional code execute the rule-based logic?
3. What programmatic checks will capture invalid outputs before they reach production?

Treating AI models as single components within a broader software system—rather than expecting them to manage entire end-to-end tasks—enables developers to build more reliable, maintainable, and cost-efficient applications.

Source: Stop Optimizing the Prompt. Design the System. — Towards AI. Written by the Apixo team from that report.