Giving a world-action model video of a new task without any action labels lifted out-of-distribution success from 25 to 56 percent, and one demonstration per task took it to 71 percent
AspireAI community · arXiv
Broad task generalization in physical environments remains a central challenge for robot learning, and the authors study it in world action models (WAMs) through two capabilities: video generalization, which predicts the visual evolution of new tasks, and action generalization, which turns those predictions into executable behavior through a video-to-action bridge. By varying video and action supervision for tasks outside the initial training set, they separate the contributions of each to task transfer. On RoboTwin 2.0, using only in-distribution video-action supervision yields a 25.07% success rate on out-of-distribution tasks. Introducing videos of those tasks without their action labels raises success to 55.90%, and adding just one action-labeled demonstration per task improves it further to 70.70%. The reading is that broader video coverage enables transfer by leveraging existing inverse-dynamics capability, and even sparse action supervision strengthens that capability further. WorldBridge is a WAM with separate video and action experts connected through structured attention and trained with joint flow matching, incorporating visual memory at two temporal scales and action forcing, alongside a unified data pipeline that standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision. The limitation is that the ablation is measured on one benchmark with a single out-of-distribution split, so the general rule that video beats action supervision for transfer is established there rather than in general.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.