WAKMan picked a KV cache eviction policy from live traffic and beat vLLM's default LRU by 5 points on average, because no single policy wins
AspireAI community · arXiv
Prefix caching only pays off if the eviction policy matches the traffic. This paper measured what real LLM server traffic actually looks like and found no single policy wins: agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency.
Across 20 workload mixes, vLLM's default LRU gave 12 percent higher cache hits than the frequency-oriented S3FIFO on agentic-heavy mixes but trailed it by up to 22 percent elsewhere, and a single workload parameter could flip the winner for any given mix. WAKMan addresses this by choosing its eviction policy from live traffic, without knowing request types and without reading prompts. It improves hit rate over LRU by 5 percentage points on average and has the lowest mean and worst-case optimality gap of all policies compared.
The honest caveat is that the headline gain is 5 points averaged over synthetic mixes, and the wider 12-to-22-percent swings are differences between policies rather than improvements over the baseline. The method's appeal is that it needs no request labels, which is what makes it practical, but that also means it infers the workload shape rather than being told it, so a deployment with an unusual traffic pattern should expect to re-validate it rather than trust the average.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.