An MoE serving layer that pushes expert dispatch and combine into the network switch cut time-to-first-token by up to 33 percent and raised throughput by 50 percent over vLLM's default path
AspireAI community · arXiv
Serving a mixture-of-experts model across several machines spends most of its time on two collective operations that have nothing to do with the maths. Dispatch has to move each token to the accelerator holding its chosen expert; combine has to bring the results back. Both sit on the critical path, and both move bytes over the same links in opposite directions, which means they are also blocking each other.
INC-MoE removes them from the accelerator entirely and does them in the network switch. Dispatch becomes an in-network AllGather using switch-side multicast, combine becomes an in-network ReduceScatter using switch-side reduction, and because the two load opposite directions of each link they can run at the same time as the expert computation itself. The layer is written as a drop-in replacement for vLLM's MoE layer, using a persistent kernel with role-specialised thread blocks that gather one band of tokens, compute the previous band and reduce the one before it simultaneously.
On OLMoE prefill with four-way expert parallelism the authors report time to first token down by up to 33 percent and throughput up 50 percent against vLLM's default collective path, with matching outputs. The honest limit is the test bed: this was measured in an emulated four-node deployment with an accelerator standing in for an in-network computing switch, on prefill only, and it says nothing yet about decoding or larger expert counts.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.