Dwell turned a vision-language model into its own world model and agent, cutting imagination compute 38 to 58 percent below a dedicated latent world model
AspireAI community · arXiv
Most agent systems use a pretrained vision-language model as the policy and a separate world model to predict outcomes, which means two large models to train and keep in sync. Dwell removes the second one: it reframes the VLM as a recurrent latent transition operator that predicts, adjusts and acts in latent space, using the model's own representations as both world model and policy.
The agent learns from real interactions and from latent imagination at the same time. Across five interactive visual environments (Sokoban, FrozenLake, a navigation task, PrimitiveSkill and EB-ALFRED), Dwell achieves the strongest overall performance among trainable methods, and the practical win is efficiency: it consistently requires substantially less computation than AdaWorld, reducing GPU-hours by approximately 38 to 58 percent across the four evaluated environments.
The framing the authors want readers to take is that effective world models for VLM agents need not reconstruct the future, they need to predict futures that are useful for action. The honest limitations: the comparison is against trainable baselines rather than the whole literature, the compute saving is reported on four of the five environments, and the results come from the authors' own implementation, so the margin over other latent-RL systems should be read as directional.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.