Google Research Introduces Sleep Framework for LLMs to Prevent Forgetting
Google Research has unveiled a sleep framework for large language models, alternating between active interaction and offline memory consolidation to enable continuous learning without catastrophic forgetting.
Modern large language models typically rely on pre-training and in-context learning, leaving them largely static after deployment. When systems are updated with fresh data, continual parameter changes often cause catastrophic forgetting, where new learning damages performance on past tasks. Alternatively, relying strictly on in-context learning limits knowledge retention to the current session. To address these limitations, Google Research has introduced a sleep framework, a learning paradigm inspired by biological memory consolidation.
Developed by researchers including Ali Behrouz from Google and Cornell University, Farnoosh Hashemi, Adel Javanmard from Google and the University of Southern California, and Vahab Mirrokni, the framework gives models a continuous periodic lifecycle. Reviewers Yuan Deng and Corinna Cortes provided valuable suggestions for the work, with support from Mark Simborg, Kimberly Schwede, and Tim Herrmann.
Active and Sleep Phases
Under this paradigm, a model operates across two distinct states. During the active or wake phase, the system interacts with inputs and temporarily stores information in fast, short-term memory components. During the offline sleep phase, the model halts new inputs and focuses on memory consolidation and self-improvement using a fixed architecture and human-defined rules.
The sleep cycle uses pre-allocated Mixture-of-Experts memory blocks and updates numerical weights via self-generated practice examples. This process is split into two core stages: memory consolidation and dreaming. Memory consolidation mimics slow-wave sleep by expanding parameters, utilizing a continuum memory hierarchy, and transferring fragile short-term memories into long-term parameters through generalized knowledge distillation and reinforcement learning. Synaptic pruning then resets the temporary parameters to free up space.
The dreaming stage mirrors REM sleep by implementing an autonomous self-improvement loop. The system randomly selects experts during synthetic data generation, creates synthetic training samples, and applies gradient-based data selection to filter the most useful experiences. Developers looking to experiment with various architectures can try top AI models cheaply through one API at https://apixoai.online.
What it means for developers
For developers and engineers working with deployed language models, this sleep paradigm offers a structured path to handle continual learning without degrading past capabilities. Theoretical analysis shows that knowledge seeding operates as a structured, low-rank compression problem, while dreaming improves the geometry of the optimization landscape by diversifying and selecting updates.
In evaluations, the sleep framework delivered strong performance across multiple domains. For continual translation in unseen languages like Manchu and Kalamang, the method maintained near-zero performance drop, avoiding the catastrophic forgetting seen in standard baselines. In long-context understanding on the BABILong benchmark, the approach scaled up to 10 million tokens, outperforming models like GPT-4 and Titans by compressing information directly into parametric memory. For factual knowledge integration on SQuAD and continued pre-training, the method outperformed base models, supervised fine-tuning, and SEAL.
Additionally, on mathematical reasoning benchmarks including AIME 24/25 and HMMT 25, the sleep-enhanced base model reached accuracy scores of 79.2%, 69.0%, and 46.1%. These results demonstrate that combining memory consolidation with imitation learning can drive superior reasoning and long-term adaptation.
Source: The sleep framework: A new way for LLMs to retain knowledge in continual learning — Google Research. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
