FP4 quantization left an accuracy gap, and storing the low-rank correction as plus-or-minus-one bits closed it at 16 times less storage
AspireAI community · arXiv (BiLoCo authors)
Post-training quantization to FP4 cuts inference cost on Blackwell GPUs with native FP4 tensor cores, but direct FP4 leaves a noticeable accuracy gap. SVDQuant closes it with high-precision low-rank corrections, which need many bits per rank and run on the same tensor cores as the main FP4 GEMM. BiLoCo instead stores the left and right factors of the correction as binary sign vectors rather than BF16 numbers, which the paper reports uses 16 times less storage per rank-one component and, because the correction becomes additions and subtractions, can run on CUDA cores concurrently with the FP4 tensor-core GEMM. On 4B to 72B LLMs it matches or improves memory-matched SVDQuant for PTQ. The implementation is a fused single kernel on B300 where the FP4 GEMM runs on tensor cores and the bit-packed correction runs on CUDA cores of the same launch, combined in the GEMM epilogue; at batch size 1 the fused kernel is at most 11 percent slower than the FP4-only GEMM for every rank up to 512, and at rank 128 at most 20 percent slower up to batch size 8. The limitation is that the speed measurements are on B300 hardware and the accuracy comparison is against memory-matched SVDQuant rather than an unquantized baseline.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.