Splitting LLM serving across a GPU and an NPU cost 7.88 percent of throughput and bought 1.26 times the power efficiency
AspireAI community · arXiv preprint
Prefill and decode want different hardware. Prefill is compute-bound and suits a GPU; decode is memory-bound and can run well on an NPU. The obstacle to splitting an inference request across both is that the key-value cache produced during prefill has to be moved to the decode device, and GPU and NPU disagree on tensor layout.
HAWK resolves that in the transfer path rather than around it. It sets up an RDMA path for direct cross-device movement and allocates tensor-layout-aware RDMA buffers on the NPU side, so a tensor arriving from a conventional contiguous GPU buffer is already in the shape the NPU expects and can be consumed with no extra rearrangement. The system was built on commercially available GPUs and NPUs and evaluated under matching service-level objectives.
The measured trade is explicit: performance within 7.88% of a GPU-only system, up to 1.26x higher power efficiency and up to 1.72x higher cost efficiency, with the decode throughput gap ranging from 7.48% on chatbot workloads to 7.88% on summarization. The honest framing is that this is not free throughput - it is a different efficiency curve for operators whose bottleneck is power or spend rather than peak speed. Caveat: the efficiency numbers come from one hardware pairing, and the layout handling is specific to it.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.