Richard Sutton and AI Researchers Question Raw Compute LLM Scaling
Turing Award winner Richard Sutton argues raw compute is insufficient for AI, triggering debate among researchers and industry leaders over the future of LLM scaling.
Turing Award winner Richard Sutton, best known for authoring the influential 2019 essay "The Bitter Lesson," has sparked widespread debate across the artificial intelligence community following a recent post on X. Sutton stated that relying purely on raw compute is insufficient to achieve true intelligence, asserting that AI science remains "far from finished" and urgently requires clever new methods. The remark has reignited discussions among researchers and tech leaders regarding whether the current paradigm of scaling large language models (LLMs) can lead to general intelligence or if entirely different architectures are necessary.
Rethinking compute and architectural limits
In his original 2019 essay, Sutton argued that general methods designed to scale with compute consistently outperform complex, human-engineered designs over time. However, his recent statements clarify that simply scaling current LLM architectures faces fundamental boundaries. Sutton has previously criticized relying on synthetic data, describing it as a "big mistake" due to the microscopic nature of simulations compared to the infinite complexity of the real world. He emphasizes that today's language models fail to continuously learn after their initial training phase and are limited by internet text, which cannot capture the full complexity of the real world. Instead, Sutton advocates for AI agents equipped with world models that learn continuously from their own direct experiences without destroying previously acquired knowledge.
Other prominent researchers have chimed in on the discussion. Former OpenAI researcher Jerry Tworek, who now runs the AI lab Core Automation, noted that while the core premise of the bitter lesson remains valid, algorithmic choices become increasingly critical as compute grows during model training. Tworek warned that many scaling dimensions remain unexplored and cautioned that the initial success of transformer scaling has effectively "reward-hacked the brains of a whole generation of scientists." Meanwhile, AI researcher Jayden Teoh highlighted how misinterpreting Sutton's bitter lesson has led the academic community to dismiss valuable research that strays from pure scaling. Teoh emphasized that intermediate or specialized ideas do not need to scale indefinitely to serve as vital stepping stones for AI progress.
The industry counter-argument for continuous scaling
Not everyone agrees that current LLM scaling strategies are approaching a wall. Leading executives at major AI companies maintain that expanding existing language model approaches still holds immense potential. OpenAI CEO Sam Altman maintains that scaling is far from exhausted, arguing that critics who assert hard limits risk discouraging researchers. Altman pointed to a recent OpenAI model that successfully disproved a long-standing mathematical conjecture as evidence of ongoing scaling breakthroughs. While acknowledging the utility of world models in fields like robotics, Altman insists that empirical data strongly favors continued LLM scaling.
Similarly, Anthropic CEO Dario Amodei has characterized scaling progress as a "rainbow with no end." Nevertheless, an emerging wave of startups is choosing to bet against sole reliance on traditional pretraining and reinforcement learning. A recent analysis by Frontier Radar #5 tracked 45 new AI research labs—including Sutton's Oak Lab and Tworek's Core Automation—focusing specifically on developing alternative learning algorithms designed to bypass the perceived limitations of current LLMs.
What it means for developers
For software engineers and system architects, this ongoing division between fundamental research labs and major commercial providers highlights the need for pragmatic AI integration strategies. While research pioneers like Sutton work on next-generation continuous learning algorithms and world models, today's applications still depend on evaluating and deploying state-of-the-art foundation models.
To stay adaptable across shifting model capabilities, developers can try top AI models cheaply through one API at https://apixoai.online, simplifying integration across different model families without locking their infrastructure into a single provider's architecture.
As alternative approaches emerge from labs exploring post-transformer paradigms, developers should avoid over-indexing on any single training methodology. Building systems with modular model layers allows engineering teams to take immediate advantage of current scaling achievements while remaining ready to integrate future breakthroughs in continuous learning and agentic world models.
Source: The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts — The Decoder. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
