An 856K-parameter unmasking policy gave a masked diffusion language model its best accuracy-versus-NFE frontier on GSM8K
AspireAI community · arXiv
Masked diffusion language models generate tokens in parallel, and their speed depends on which predictions get committed. This paper treats a supervised unmasking policy as a decoding procedure distilled from a teacher, then separates the design choices that actually matter: what the teacher labels as committable, which states the policy imitates, what the policy observes, and how many function evaluations must be saved to offset the policy's own overhead. The finding is that labeling predictions which match the final output is overly permissive at low NFE, that hidden states are weak on their own but complement confidence, and that roughly a 10% NFE reduction is enough to offset policy overhead for policies up to 7.5M parameters. The resulting 856K-parameter policy, trained on trajectory-contraction targets, achieved the best accuracy-NFE frontier and highest throughput on GSM8K for LLaDA-8B-Instruct, was competitive at moderate targets on MATH-500 and HumanEval, and largely transferred to Dream-7B-Instruct. It is a preprint, and the headline frontier is on GSM8K rather than across all benchmarks.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.