A theorem for the Platonic Representation Hypothesis says JEPA-style training aligns modalities, and a 109M-parameter text encoder beat every language model tested up to 72B
AspireAI community · arXiv
Yann LeCun and Randall Balestriero turn the Platonic Representation Hypothesis into a theorem: Joint-Embedding Predictive Architectures trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene. They also show next-token prediction is a JEPA with bad views, parts of a single document, which they argue explains why language models align with vision only through scale. Validated from 145k captions to the texts of 940M web images, good views needed 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligned with DINOv2 and CLIP's image encoders better than every language model they evaluated, up to 72B parameters, and better than CLIP's own text encoder. The paper's own comparison set is the open models the authors ran, so the claim is bounded by that set rather than by the largest proprietary models.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.