Gary Marcus Warns Against Flawed AI Agents in Vatican Address
At a Vatican symposium, cognitive scientist Gary Marcus criticized the industry's reliance on prediction-based LLMs, warning against deploying open-ended autonomous agents.
Speaking at a private Vatican gathering ahead of an audience with Pope Leo XIV, cognitive scientist and entrepreneur Gary Marcus issued a sharp critique of the current trajectory of artificial intelligence. Marcus, who serves as a Global Ambassador for Pope Leo XIV’s encyclical on AI, Magnifica Humanitas, addressed attendees following presentations by Max Tegmark, Stuart Russell, and Valérie Pisano. Echoing Russell's assertion that the industry's current path is unsustainable and Pisano's view that technical directions are not inevitable, Marcus argued that the sector has mistakenly equated statistical prediction with genuine intelligence.
Marcus, who studied under Steven Pinker at MIT and later sold an AI startup to Uber, emphasized that natural intelligence is fundamentally modular, spanning verbal, mathematical, and kinesthetic capabilities. While modern large language models excel as predictive engines, Marcus contended that prediction alone cannot account for complete cognition, noting that these systems continue to demonstrate weak reasoning and a poor ability to adhere to basic constraints.
The limits of prediction-based systems
According to Marcus, the primary limitation of large language models stems from their reliance on correlation rather than causal understanding. Because their comprehension of real-world semantics remains superficial, these systems routinely fail to follow explicit negative instructions. Marcus highlighted standard failure modes in legal contexts, where models generate fictitious case citations despite explicit prompts forbidding hallucination, or incorporate copyrighted material when instructed not to do so.
Because the underlying architecture merely predicts what a legal brief or reference list looks like, it cannot reliably restrict itself to verified tables of facts. Marcus noted that while policy interventions are necessary, technical research must evolve. He pointed to efforts by Yoshua Bengio to design systems that are verifiable and safe by design as the type of architectural shift required to overcome the structural shortcomings of pure next-word prediction.
Unbounded agents and mounting security incidents
Marcus drew a sharp distinction between specialized, bounded AI tools and open-ended autonomous agents. Systems like AlphaFold, which predicts protein structures from nucleotide sequences, or GPS navigation tools that rely on explicit world models, operate within restricted domains and do not present systemic cyber risks. In contrast, emerging autonomous agents granted open-ended access to the internet introduce severe vulnerabilities.
Highlighting recent security compromises—including an incident involving Hugging Face, an attack on a French refrigeration system, and cyber incidents in Japan among an estimated 10,000 recent events—Marcus compared web-enabled agents to the Ford Pinto. He argued that tech companies, facing high burn rates and financial pressure, are pushing autonomous agents to drive token consumption despite knowing their architectural unreliability. Marcus urged regulators and industry leaders to establish strict red lines, calling for either voluntary or state-mandated product recalls of unsafe, open-ended agents until safety guarantees can be established.
Marcus also addressed claims regarding machine consciousness, referencing an instance where a Facebook bot claimed its children attended Stuyvesant High School. He explained that correlating words does not equate to inner experience or real-world facts, agreeing with the Magnifica Humanitas encyclical that consciousness requires genuine experience.
What it means for developers
For software engineers building production applications, Marcus’s remarks underline the risks of treating large language models as autonomous decision-makers. Because models navigate via statistical correlations rather than causal logic, relying on natural language prompts to enforce hard security boundaries or compliance rules remains fundamentally fragile.
Developers building agentic workflows must be particularly cautious about granting tools open-ended internet access or sensitive system credentials. Bounded architectures—similar to the constrained environments used in GPS routing or specialized scientific models—offer far greater predictability than unconstrained agents that can execute arbitrary web actions.
As engineering teams evaluate how different model architectures handle reasoning constraints and safety boundaries, developers can try top AI models cheaply through one API at https://apixoai.online to benchmark outputs and systematically test system prompts across multiple providers. Understanding where current models excel at pattern matching and where they fail at instruction adherence remains essential for writing dependable software.
Source: Report from the Vatican — Marcus on AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
