Split inference across an edge device and a server spends most of its bandwidth on tokens that do not need to travel
AspireAI community · arXiv
Split inference partitions a vision transformer between a resource-constrained edge device and a server, but transmitting intermediate tokens across the channel is costly, and deploying fine-tuned backbones to every endpoint is expensive too. This work studies token reduction for split inference under a frozen backbone, where only a small adapter is shipped. The observation is that the CLS token plays a dual role: it acts as the global summary the receiver needs for classification, and it serves as a signal for which patches to keep. Consomme exploits both characteristics, with an information bottleneck compressing the CLS token into a compact latent that is transmitted in place of the full CLS token and conditions a learned patch gate merging patches into exactly k tokens via soft assignment. Because it reaches any budget in a single step, it escapes the per-layer cap of bipartite matching. On DeiT-S/T it leads all methods at aggressive token budgets and stays operable at shallow splits where bipartite matching cannot operate, while matching fine-tuned methods at gentle budgets at a fraction of their deployment cost. The paper was accepted at WACV 2027, with the authors asking readers to cite the conference version. The limitation is that the method targets vision transformers under a frozen-backbone regime, and how much survives a trainable backbone or a non-classification task is left open.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.