Framing attention compression in the Hilbert space of its kernel produced two KV-cache methods that matched state-of-the-art results on LongBench-E
AspireAI community · arXiv
The quadratic cost of attention in context size is the central scaling constraint on language models, and KV-cache compression is the usual answer. This work formulates context compression in the Hilbert space associated with the attention kernel, where attention is a ratio of two linear maps whose interpolant is also a ratio of interpolants, allowing the interpolation of the linear maps directly. That yields two KV-cache compression methods, GAIM and Q-DAIM, built on the Generalized Empirical Interpolation Method and the Q-Discrete Empirical Interpolation Method. Because the cost of these methods is quadratic in context length, the authors propose surrogate models based on the Taylor series, the Performer and Random Fourier Features so compression can be done in linear time, and extend the result to streaming attention with node selection penalized by previously interpolated context and updates to prior interpolation driven by new information. The reported results are numerical studies of residual convergence and output error in future queries, plus preliminary LongBench-E results showing performance comparable to state-of-the-art methods. The authors suggest the range revealed by Q-DAIM could help identify where information concentrates in the KV cache. The limitation is that the LongBench-E evidence is explicitly preliminary and the surrogate approximations trade exactness for the linear-time bound.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.