SPOT, a 0.62M-parameter model, cut target-model passes by up to 6.8x and reached 3.4x the throughput of standard decoding on SDAR-8B by predicting which position a diffusion language model commits next
AspireAI community · arXiv (Technion / AI Labs)
On the SPOT, an arXiv paper submitted October 15, 2026, identifies why speculative decoding has not carried over to diffusion language models: those models must choose both what token to generate and where to generate it. The authors show that the model's own predictions from the previous pass already contain the right token at more than 90% of positions, but not which position comes next. SPOT - Speculating Positions Order Trajectory - is a 0.62M-parameter model that reads only the target's output distributions, predicts that order and proposes a tree of candidate continuations, which the target verifies in a single pass by state equality; shared Gumbel noise between drafter and target keeps verification exact at any temperature. The authors report that SPOT cuts target passes by up to 6.8x and reaches up to 3.4x the throughput of standard decoding on SDAR-8B, matching or beating calibrated and training-free drafting baselines in every task and budget at T=0 and T=1, with the learned drafter transferring across six different diffusion language models. SPOT-Sets, the dynamic-threshold extension, improves SDAR-8B throughput by 59% on average over the threshold decoder alone. The limitation is that the speedups are measured on diffusion language models specifically, a family with far smaller reader adoption than autoregressive ones, and the draft-verify approach still needs the target model's full forward pass.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.