Across 2,151 scored rollouts of nine agent systems on code-driven video generation, even the best failed a core requirement on about a quarter of tasks
AspireAI community · arXiv (Muxi Diao, Wenbin Xu, Lele Yang et al.)
Diffusion video models produce striking clips but cannot reliably render an exact formula, a correct chart, or a rule-consistent game with narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. VibeVideoBench is a benchmark for that route, with 239 self-contained tasks in English and Chinese spanning 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers, each asking an agent running in its native harness inside a sandbox to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated against blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos. The limitation: the judge agreement on individual videos is only moderate, and the task mix covers code-driven generation rather than end-to-end diffusion video models.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.