DISCO removed the vocabulary-scale handshake from distributed speculative decoding and still held about 1.5x speedup at 1 Mbps bandwidth
AspireAI community · arXiv
Distributed speculative decoding lets a small model on-device draft tokens that a server-side large model verifies, but exact rejection sampling normally requires shipping the full token distribution over the LLM vocabulary between device and server, and that communication cost eats the speedup on bandwidth-constrained links.
DISCO removes that transfer while preserving the exact target distribution. The trick is a server-side replica of the draft small model, so the server can reconstruct the draft-side distribution locally and only the drafted token indices and scalar probability values cross the wire. In experiments on GSM8K, CNN/DailyMail and HumanEval, with bandwidth varied from 1 to 100 Mbps, RTT fixed at 20 ms and draft length from 2 to 8, DISCO maintained roughly 1.5x speedup even under extreme bandwidth restrictions and more than 1.3x for both model families at 1 and 5 Mbps. The communication term scales roughly with 1/BW while compute stays nearly constant, with about 1.5 ms per token of extra server-side resampling.
The honest limitation: the speedup is measured against two specific baselines on a simulation of network conditions, and the server-side draft replica trades bandwidth for memory and a little extra latency. It is a systems paper about device-server collaboration, not a general inference stack change, so readers running their own fleet will need to weigh the replica's cost against their link.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.