Google's sleep framework gives LLMs a wake-and-rest cycle, retaining long-context knowledge up to 10M tokens without catastrophic forgetting
AspireAI community · Google Research
Google Research introduced a continual-learning paradigm on October 27, 2026 that gives a deployed language model an alternating wake-and-sleep lifecycle, so it can keep absorbing new facts instead of freezing at a knowledge cutoff date. During the wake phase the model handles input and parks fragile observations in short-term memory; during the sleep phase it unlocks pre-allocated Mixture-of-Experts blocks and consolidates that short-term trace into stable long-term parameters, then resets the temporary memory. Consolidation uses Generalized Knowledge Distillation plus reinforcement-learning imitation so the model keeps its own on-policy behavior, while the dreaming stage activates non-standard experts to generate synthetic practice data, filters it by gradient-based selection and fine-tunes on the survivors. Reported results include near-perfect BABILong long-context scaling up to 10M tokens, 48.9% on single-task factual incorporation and 46.2% under continued pre-training on SQuAD, and AIME 24, AIME 25 and HMMT 25 accuracy of 79.2%, 69.0% and 46.1%, with near-zero forgetting when learning continual translation in Manchu and Kalamang. The honest limitation is that this is a research prototype with a fixed architecture and human-defined training rules, not a shipping product, and the reported comparisons are against in-context learning, Cartridges and supervised fine-tuning rather than against other deployed continual-learning systems.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.