Vision models read the scene in front of them well, and that turns out to be a different thing from remembering it.
AspireAI community · arXiv
**Vision models read the scene in front of them well, and that turns out to be a different thing from remembering it.**
The question was whether a visual foundation model maintains a spatial state that persists, updates under self-motion and guides action, rather than recognising individual frames. The authors probe ten frozen representations, including DINOv2, V-JEPA2, VideoMAE, Fast3R, Wan and Qwen2.5-VL, at four levels: spatial structure, object persistence, predictive spatial update and actionability.
No representation led on everything. Fast3R had the strongest camera-pose and object-persistence readouts, DINOv2 led future-feature prediction, V-JEPA2 led path integration, and VideoMAE localised previously seen objects better than DINOv2 and Qwen2.5-VL despite weaker depth readouts. Object persistence tracked camera-pose readout, not visible-scene metrics, and longer prefixes made persistence worse for all eight models tested. Matched controls separated decodability from input use: for V-JEPA2, copying the last feature beat learned prediction, and visual history helped V-JEPA2 and Wan but not VideoMAE. And input use was not utility: changing the goal rotated predicted actions by 30 to 46 degrees, yet the right goal did not reliably improve progress.
The honest limit: a probing study on frozen features over static indoor scenes, with no object dynamics.
https://arxiv.org/abs/2610.40636
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.