LLM-written summaries kept at most 16 percent of the details agents needed, while mechanical extraction kept 79 percent
AspireAI community · arXiv (Ruopei Sun, Jianfeng Cai, Kangwen Zhao et al., University of Science and Technology of China)
Compact-Bench isolates the cost of one specific event in an agent's life: the moment its history is replaced by a summary. It pairs full-history continuations with summary-only continuations from a shared repository state, and covers 35 SWE-bench repairs, 146 agentic sessions and 106 production compaction chains.
Across 14 summarizer backbones, the damage is not subtle. Stable damage ranges from 5 to 64 percent of the 22 full-history successes, Claude Opus 4.8 preserves the most repairs and DeepSeek-Flash damages the most. With Kimi-K3 as the summarizer, the Kimi Code prompt preserved the most task continuations at 19 of 22, while the Claude Code prompt retained the most recurring details at 15.0 percent.
The headline comparison is the one to act on: LLM summaries retained at most 16.0 percent of recurring details, against 79.1 percent for mechanical extraction. Production summaries retained 15.7 percent at the first boundary. Failures also recur rather than scatter, with one repair damaged by 13 of 14 backbones and another by all 5 prompts.
Caveat: 22 full-history successes is a small base, so the per-task percentages are wide. The extraction-versus-summarization gap is the part that replicates.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.