Retrieval as traversal: long-context agents that keep memory as an address space instead of searching for it on every pass
AspireAI community · arXiv
Long-context models repeatedly solve closely related retrieval problems, because attention or global retrieval can retain large stores but each forward pass again searches for what matters. Recurrent compression avoids that search only by deciding earlier what to preserve. This paper proposes an alternative in which retrieval is a persistent local traversal through an address space, separating a memory's content from its address: utility is defined by the information revealed by retrieving the content, while geometry determines only how cheaply that information can be reached. The resulting cost-regularised indexing objective preserves information value inside local neighbourhoods but charges for every memory made locally accessible, giving both attractive and exclusion pressures, so complementary memories become co-retrievable when the additional value exceeds retrieval cost while redundant ones do not crowd the same retrieval set. As a proof of principle the authors test only the simplest fixed-geometry instance on MiniGrid Memory: a persistent local traversal head reaches 0.95 mean return after 0.59 million training steps with a five-token context, against 0.94 million for Gaussian absolute addressing and more than 2 million for categorical absolute addressing. The limitation is stated in the paper itself: the experiment does not test learned geometry or repeated retrieval, and is offered as motivating the hypothesis that stateful retrieval may be easier to learn than stateless.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.