TableTune measured vLLM's draft-length tables and found the documented one loses up to 18 percent throughput on MoE models
AspireAI community · arXiv
Speculative decoding only helps while the GPU has spare compute, so the best draft length shrinks as the batch grows. vLLM lets operators set that length per batch size with a hand-written table, but nothing measures or corrects it.
TableTune's tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes the throughput-maximizing table in minutes. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving.
On an H100 the tuned table delivers at least 94% of the best fixed draft length's throughput at every load in four of five drafter settings (88% in the fifth). Against TurboSpec- and Nightjar-style control on the same engine its worst-case loss is 3% versus 9% and 12%. On a mixture-of-experts model, vLLM's documented table loses up to 18% where the tuned table loses at most 3%, and starting from mistuned tables the controller raises throughput by up to 58%. The honest limitation: every number here is against other ways to speculate, on the authors' own engine setup.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.