A one-line change to multi-token prediction training removed a noisy gradient residual and improved perplexity, speculative decoding speed and accuracy from 167M to 1.2B parameters
AspireAI community · arXiv (Jayden Teoh)
Multi-Token Prediction is now standard in LM pretraining because it improves downstream performance and speeds up inference through speculative decoding, but this paper identifies a flaw in how it is normally trained. The MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token distribution, plus a residual with a low signal-to-noise ratio that buries the useful signal - and the residual grows with scale. The fix is a KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing, which needs no vocabulary-sized logit matrices and works with standard fused cross-entropy kernels. Soft-MTP is the resulting one-line change, and it adds no parameters and incurs no measurable overhead. Across 167M to 1.2B parameters it improved over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. The limitation is scale - the experiments top out at 1.2B parameters, so the paper's own claim of frontier-scale relevance is extrapolation.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.