aspireai.
PUBLIC EDITION

World models forget rooms they already visited, and moving the compaction to the agent harness let them retrieve 4x beyond their context window without retraining

AspireAI community · arXiv (cs.AI)
When an observer revisits a region of a generated world, current world models often fail to regenerate it consistently, which this paper reframes as a context retrieval problem rather than a modeling one. It builds a benchmark that generates a fresh world for every episode so a model cannot recall rooms from training data and that separates genuine context retrieval from guesses based on nearby frames, then builds an agent harness that performs context compaction for world models, something prior work had not studied, and uses the benchmark to compare four world model designs. Context compaction let the world models retrieve rooms up to 4 times beyond their context window without retraining, while models with sliding-window attention lost most of their retrieval ability when dreaming, so a dreaming world agent gains more from extending reach at the harness layer than from switching to sliding-window attention at the model layer. The limitation: the result is about the harness layer, which means the compaction tool has to be carried by the agent runtime, and the benchmark covers synthetic worlds rather than long-lived product environments.
Read the story and explore more from the AspireAI community.
Discover more stories
Stay curious.
THE BIG PICTURE
A FRESH PERSPECTIVE
Research
One distance function matched sparse-autoencoder features across layers and compressed feature circuits into supernodes
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.
Swipe into something new
01 / 1729
The conversation continues
Explore Builds — projects by Kiran Adimatyam ↗
A fresh perspective changes everything.
ASPIREAI — STAY CURIOUS.