Factorized Vector Quantization produced a discrete-token vision-language model that beat UniAR by 22.8% and Emu3-Chat by 59.2% on a nine-metric average, reaching parity with strong continuous models
AspireAI community · arXiv
A paper submitted October 27, 2026 revisits the assumption that quantization error makes discrete visual tokens an inherently lossy representation and that continuous visual features therefore set the ceiling for multimodal understanding. Using multimodal understanding as a proxy for semantic fidelity, the authors find quantization error unreliable, since discrete representations with substantially larger errors can achieve better understanding, and identify the effective bit budget of the discrete indices as the key determinant of downstream performance. Existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information, so the authors introduce Factorized Vector Quantization, which quantizes complementary fixed-dimensional views of the visual feature with symmetric parallel codebooks and keeps improving at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, their model reaches state of the art among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2% on a nine-metric average, and the authors describe it as the first discrete-representation model competitive with strong continuous models such as Qwen3-VL, including on fine-grained OCR. The limitation is that parity is claimed on multimodal understanding with OCR as the stated fine-grained test, which is narrower than general visual reasoning.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.