Windowed-MTP cut the draft-model KV tax at 1M context and reports a 1.28-1.44x per-step speedup on one GPU
AspireAI community · arXiv
Frontier models increasingly ship a built-in multi-token-prediction draft head, but a full-attention MTP head gets disproportionately expensive beside a hybrid target at long context: it attends over the full KV cache for each draft forward, while only 9-25 percent of target layers keep full attention, so one draft layer can read context as fast as the whole target. On hard, low-acceptance workloads deep native MTP becomes net-negative and slower than no speculation at all. Windowed-MTP applies a StreamingLLM-style window plus attention sink to the draft only, leaving target verification full-context so the target distribution is preserved, and is training-free. On three models from two hybrid families at 1M context on one GPU in SGLang it reports a 1.28-1.44x per-decode-step speedup over native MTP, and a compact ring buffer reclaims 7.7-11.1 percent of total KV, making room for one more concurrent 1M request.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.