A closed-loop 3D reasoning benchmark scored 10 frontier MLLMs from 4.8 to 79.1, and 46 percent of their failures were collisions or observation loops
AspireAI community · arXiv (Shi, Yuan, Song, Pu, Wu)
To solve problems in 3D space rather than merely describe images, multimodal LLMs need to reconstruct geometry from occluded views, anticipate dynamics and execute precise closed-loop actions - which prior studies avoid by testing passive VQA on static images or supplying privileged coordinates. SpaceAct is an interactive benchmark built on interlocking bricks and mechanical puzzles with 108 verified instances across 36 task families in six categories, no privileged coordinates, and diagnostic modes including a reference plan, exact 3D state, no-camera, blind and a seven-lever variant. Models rotate the camera to resolve occlusion, read in-scene rulers, output 6-DoF poses under swept-collision and statics checks, and self-correct through split-screen rollback feedback. Across 10 frontier MLLMs on the 44 hardest instances, scores span 4.8 to 79.1, with seven models solving at most 27.3 percent of 6-DoF tasks. For the top three, a reference plan closes 44 percent of the gap while leaving 5.2 collisions per episode, whereas adding exact 3D state closes 90 percent and cuts collisions to 0.5; for four others a plan lifts the mean from 10.9 to 35.3 and exact state to 81.8. The limitation is that the physical puzzles are a narrow domain, and 46 percent of failures being collisions or observation loops says the benchmark is currently measuring perception and actuation as much as reasoning.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.