A 3B-parameter text-to-image model trained for 6,600 H100 GPU-hours reached 86 on GenEval and 87 on DPG-Bench by splitting semantic learning from rendering
AspireAI community · arXiv
A paper submitted October 27, 2026 proposes separating the two jobs visual generation demands, using autoregressive pretraining for knowledge acquisition and diffusion post-training for high-fidelity rendering. Controlled comparisons identify autoregression as the stronger pretraining objective for downstream rendering, improving both learning efficiency and final generation quality, and the authors find that AR-only generation quality does not necessarily correlate with final performance because semantic quality matters more. That finding motivates a purely semantic tokenizer trained with language supervision alone, without an image decoder or reconstruction loss, and the paper establishes compute-optimal scaling laws for the knowledge-acquisition and rendering phases separately, with autoregressive pretraining lowering the fitted asymptotic rendering loss against pixel diffusion models trained from scratch. Guided by those laws the authors trained a 3B-parameter text-to-image model using only 6,600 H100 GPU-hours and reached 86 on GenEval and 87 on DPG-Bench, validating the paradigm at scale. The limitation is that the compute figure and the two scores describe one architecture at 3B parameters, so it does not establish that the split beats diffusion-only training for every model size.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.