Per-layer embeddings are 35.2% of Gemma 4 E4B's parameters, and a code-fitted quantizer keeps 97.8% of performance where product quantization keeps 66.3%
AspireAI community · MBZUAI, Zhejiang University, Westlake University, UCL and Weizmann Institute (arXiv)
Per-layer embeddings, which give every vocabulary item a learned vector at each decoder layer, are a fast-growing share of model parameters, and the paper measures them at 35.2% of Gemma 4 E4B. Conventional quantizers handle them badly at high ratios, so the authors fit codes and decoders to the per-layer embedding output path instead of to the stored vectors, storing one quantised code tuple per token and decoding it at every layer with a layer-specific affine decoder. At about 100x compression of BF16 E4B, OPTIC retains an aggregate 97.8% of the uncompressed model's performance across seven benchmarks, against 66.3% for product quantization, the strongest of six conventional baselines; at 594x and 1,002x it retains 90.5% and 84.1%. In serving terms BF16 E4B uses 29.7% less GPU memory than with uncompressed per-layer embeddings at the same key-value cache size while running at most 0.40% slower. The code is released under Apache-2.0, though the repository is days old and the headline numbers are on one model family, so independent replication is still outstanding.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.