Engineering Leaders Outline Operational Risks and Governance for AI in Production
Senior engineering practitioners examine silent AI failures, the limits of manual code review, and the need for persistent evaluation and governance in production agent systems.
Deploying artificial intelligence systems into production environments creates operational headaches distinct from traditional software engineering. During an InfoQ panel titled "AI in Production: What Breaks, What Works, and Who Approves It?", moderated by Hien Luu, senior technical practitioners examined why production agents break silently, how rapid code generation overwhelms review pipelines, and how engineering teams should structure agent governance.
Luu framed the discussion around three emerging realities: AI systems fail by producing convincing errors rather than throwing classic runtime exceptions; automated coding tools output software faster than humans can inspect it; and sensitive information faces new vectors for exposure across prompts, context windows, and retrieval layers.
Silent Failures: Semantic and Embedding Drift
Unlike deterministic applications where errors frequently manifest as crashes or broken traces, AI pipelines often fail quietly while underlying infrastructure metrics show healthy statuses. Zichuan Xiong, Head of AI Ops for Managed Services at ThoughtWorks, detailed the problem of semantic drift. In text-to-SQL workflows, models often homogenize domain-specific terminology—such as treating the word "member" identically across separate sales and support contexts—producing skewed analytical results that bypass standard monitoring checks.
A related challenge occurs at the vector level. Financial AI practitioner Lan Chu described an incident involving embedding drift within a compliance system designed to flag prohibited investment advice using similarity thresholds. When the underlying embedding model received an unannounced background update, the vector distance distribution shifted unexpectedly. The system suffered severe spikes in false positives and false negatives, even though all traditional infrastructure checks remained green.
Security and data exposure present additional challenges. Katharine Jarmul, a privacy specialist on the panel, noted that models retain training data in ways that can be exploited to de-anonymize data or uncover API keys, warning that prompt engineering alone is inadequate for defense. Prem Chandrasekaran, Tech and Business Services Lead at ThoughtWorks, observed that retrieval-augmented generation (RAG) pipelines can mistakenly pull confidential documents, while also pointing to the GitSpawn vulnerability, which allows unauthorized code execution via malicious repository setups during standard commands.
Rethinking Code Reviews and Living Evaluations
As coding assistants generate high volumes of code, manual line-by-line inspection has become impractical. Chandrasekaran suggested shifting routine stylistic and syntax verification directly into automated CI/CD pipelines, leaving human reviewers to focus strictly on architectural integrity, business intent, and non-functional requirements. Xiong recommended shifting evaluation left by checking developer intent at the test harness level before pull requests are even opened.
To ensure tests generated in AI workflows are rigorous, Chandrasekaran highlighted mutation testing. By injecting intentional bugs into the codebase, teams can determine whether their automated suites validate core business logic rather than merely satisfying superficial code coverage figures.
For agentic systems, Chu argued that evaluations cannot remain one-time milestones completed prior to launch. Instead, teams need living evaluation infrastructure using span-level tracing, log filtering, and model-assisted categorization of failure patterns. This setup allows production edge cases to continuously inform and refresh offline testing benchmarks.
What it means for developers
For software engineers and platform architects, operating AI agents demands treating non-deterministic models with the same operational rigor applied to untrusted external integrations.
Chandrasekaran recommended managing autonomous agents with policies similar to onboarding new engineering personnel. Teams should assign agents distinct non-human identities, enforce tiered permissions that separate read-only queries from reversible write tasks, version third-party Model Context Protocol (MCP) dependencies, and isolate API secrets within sandboxes outside the agent's immediate runtime.
Jarmul emphasized adopting continuous threat modeling, systematic red teaming, tight functional boundaries, and expanding observability pipelines to capture larger event streams. Developers experimenting with different architectures and evaluating multiple providers can test top AI models cheaply through one API at https://apixoai.online to establish baseline metrics across various model families.
Ultimately, the panel concluded that while engineering, security, and privacy teams must establish technical safeguards and monitoring infrastructure, deciding when an autonomous system is ready to launch remains a broader business decision.
Source: AI in Production: Panellists Highlight Silent Failures, Harness Engineering, and Agent Governance — InfoQ AI & ML. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
