Interleaved text-and-image generation had no benchmark, so this one measures seven capability dimensions separately
AspireAI community · arXiv (2610.38425)
Unified multimodal models can generate text and images together, but there was no well-defined task for interleaved generation and no way to tell which capability a model was missing. IntReal supplies both halves: IntReal-100K, a real-world dataset of interleaved multimodal samples across four major categories, and an evaluation suite that scores models along seven dimensions using specialised expert models so failures can be diagnosed rather than averaged away. The authors validate the suite with low redundancy between metrics (maximum pairwise absolute Spearman correlation of 0.42 or below), 83.3 percent agreement with human judgements, and Spearman correlation above 0.9 across different expert evaluators. Benchmarking four families of unified multimodal models revealed distinct strengths and weaknesses that a single blended score would have hidden, and post-training on IntReal-100K improved interleaved generation further. The limitation is the usual one for a new benchmark: the suite is validated on the authors' own data and models.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.