A streaming episodic memory for vision-language navigation lifted physical success from 17 to 40 percent, and post-training pushed it to 47 percent on held-out episodes
AspireAI community · arXiv (cs.CV)
A vision-language agent walking through a 3D scene has to remember where it has been and what it saw, and most streaming agents lose that thread as the context grows. MemVLN keeps a bounded bank of compressed perceptual features, each stamped with a timestep and an action-derived relative pose, and turns them into compact memory tokens through cross-attention on top of a frozen pretrained geometry encoder. The memory module is trained with a masking-based trajectory summarization task that decomposes routes at multiple granularities, so the model has to reconstruct a route from memory alone.
On standard VLN-CE benchmarks, with training restricted to the official R2R and RxR trajectories, the paper reports state-of-the-art performance on most reported metrics. The more concrete number comes from physical evaluation: 30 instruction-conditioned episodes in one real scene, 24 used as demonstrations and 6 held out. On those held-out episodes, success goes from 17 percent for the VLM baseline to 40 percent with MemVLN, and to 47 percent after post-training.
The limitation is the size of the physical test - one scene, 30 episodes, 6 held out - so the headline progression rests on a very small sample even though the benchmark numbers are broad. The memory is also trained with a purpose-built masking objective rather than end to end, so it is a designed module, not something that emerged from general instruction tuning.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.