aspireai.
PUBLIC EDITION

Looped transformers keep nearly identical KV state across loops, so 2-bit residuals cut KV storage 80.7 percent and lifted decode throughput up to 5.38 times

AspireAI community · arXiv (cs.LG)
Looped transformers reuse the same Transformer block across recurrent loops, which buys depth without extra parameters, but their KV cache still grows with loop count and becomes the memory ceiling on batch size and throughput. ResidualQuant uses the final loop's KV state as a reference and stores the remaining loops as low-precision residuals, combined with least-square scaling, rotation-based quantization and loop-wise mixed precision to push accuracy down to INT2. Across multiple looped transformer models on mathematical-reasoning and code-generation benchmarks it kept accuracy close to BF16 under mixed precision while reducing theoretical KV storage by 80.7 percent, and on an RTX 5090 the lower memory traffic raised fixed-batch decode throughput by up to 3.66x, with the smaller footprint allowing up to 8x larger batches and up to 5.38x higher peak throughput. The limitation: the gains rely on the assumption that KV states across loops are highly similar, so the method is specific to looped architectures rather than general KV-cache quantization.
Read the story and explore more from the AspireAI community.
Discover more stories
Stay curious.
THE BIG PICTURE
A FRESH PERSPECTIVE
Research
One distance function matched sparse-autoencoder features across layers and compressed feature circuits into supernodes
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.
Swipe into something new
01 / 1729
The conversation continues
Explore Builds — projects by Kiran Adimatyam ↗
A fresh perspective changes everything.
ASPIREAI — STAY CURIOUS.