Frozen video foundation models did encode intuitive physics, but where it was readable depended far more on the readout and benchmark than on the pretraining objective
AspireAI community · arXiv
The paper asks whether video foundation models encode a usable sense of how the physical world works and whether intuitive physics emerges naturally from large-scale video pretraining. Using frozen-feature probing on IntPhys2 and Minimal Video Pairs, the authors compare predictive joint-embedding models (V-JEPA), masked reconstruction models (VideoMAE) and diffusion-based video generators (LTX-Video) across linear, MLP and temporal attentive probes. Physics-relevant information is generally weakest in early layers and most accessible at intermediate to late depth. On MVP, temporal attentive probes substantially outperform linear readouts, with V-JEPA reaching 95.0% pair consistency; on IntPhys2, linear probes recover more signal and VideoMAE attains the strongest temporal attentive result at 73.9% violation-of-expectation accuracy. On both benchmarks, attentive probe performance on LTX consistently underperforms the other two pretraining paradigms. The conclusion is that intuitive physics is broadly decodable from video foundation models but is not a single uniform capability, and its accessibility, especially its temporal grounding, depends much more on the benchmark and readout than on a simple hierarchy of pretraining objectives or model scale. The limitation is that the work measures decodability from frozen representations, not whether a model can actually use that information to predict physical outcomes.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.