A hybrid LLM serving system picked an Attention-or-SSM layout per request and cut time-to-first-token by up to 80.9 percent against vLLM
AspireAI community · arXiv
Hybrid language models mix Attention layers with state space model layers, and the usual design fixes the layout once, so every request pays for the same mixture no matter what it actually needs. This paper co-designs the model and the serving runtime so that during prefill a gate picks, per request and per hybrid block, whether that block runs Attention or the SSM, and the choice stays fixed for the rest of that request's decoding.
Evaluated on an NVIDIA H100 across six reasoning, QA and summarization datasets, the hybrid model took the best result on 13 of 18 quality metrics while cutting estimated TFLOPs per token by 40.5, 33.2 and 67.2 percent against Attention-only Llama models at three scales, and by 28.4, 28.9 and 58.7 percent against Nemotron-H. The system-level results are the headline: serving throughput improved by up to 3.87 times over vLLM, time-to-first-token dropped by 80.9, 78.9 and 75.2 percent on average against vLLM, SGLang and Jenga, and end-to-end latency fell by 54.6 to 66.0 percent on average. Hybrid-aware context caching lifted the prefix hit rate to 43.4 percent from 18.7 percent, and at 20 articles it still held 24 percent where conventional caching fell to 7 percent. Across 500 ARC-Challenge requests the system activated 35 distinct layouts, with the six most frequent covering more than 90 percent of requests.
Two honest wrinkles. The architecture and the runtime are co-designed, so this is a system where the model and the server have to change together rather than a drop-in runtime patch. And the paper's own artifacts disagree on the system's name - the arXiv abstract names it Morph while the HTML rendering names it Flex throughout - so the name should be treated as unsettled. The gains are also self-reported on the authors' own hardware.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.