Reference-overlap metrics could not tell accepted video captions from rejected ones: BLEU-4 scored an AUC of 0.510 against human judgement on dense multi-window descriptions
AspireAI community · arXiv (human correction as video-caption evaluation)
Video captioning is increasingly scored with metrics built for single-sentence image captions, and this paper asks whether those metrics recover human judgement on dense, multi-window video description. Six frontier models captioned 196 short clips and trained annotators post-edited every caption against a failure taxonomy fixed before annotation, once as a whole-clip overview and once window by window, with a quality-control reviewer checking each one, producing 1,144 overview verdicts and 4,672 window verdicts. Reference-overlap metrics fail to separate accepted from rejected captions, with BLEU-4 reaching an AUC of just 0.510 and a CIDEr-style tf-idf cosine 0.528, and restricting references to the reviewer corrections leaves BLEU-4 at 0.479, indistinguishable from chance; review granularity also changes what gets detected, with camera and shot errors on 2.7 percent of captions read as an overview against 12.8 percent read window by window, a 4.7x gap that survives every robustness check. The limitation is that the corpus is only 196 short clips and only 4 of 15 pairwise model gaps survive Bonferroni correction, so the finding is about metric failure rather than a definitive ranking of the six models.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.