GAAC compressed the KV cache 1.4x with zero loss by exploiting exponent redundancy, and cut resident KV memory about 26 percent while raising throughput 13-29 percent
AspireAI community · arXiv (GAAC authors)
Weight-only quantization shrinks the model but the key-value cache keeps growing, and on personal devices that cache is the real long-context bottleneck. GAAC is a lossless codec and serving system that encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges, adapting code widths across small groups while preserving KV values bit-for-bit, with a fused kernel that reconstructs tiles on the fly instead of decompressing the whole cache. Across five models it achieves 1.405-1.407x lossless BF16 KV compression, accelerates decode attention 1.19-1.27x at 64K context on NVIDIA DGX Spark and reduces resident KV memory about 26 percent for 13-29 percent higher aggregate throughput at comparable memory budgets; on Apple M4 it shows 1.02-1.25x attention speedups. The compression ratio is modest because the approach refuses to approximate.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.