aspireai.
PUBLIC EDITION

An audit of seven commercial LLM routers found none beat always calling one mid-priced model on quality and cost together, and five of 17 settings were just a knob in front of a constant

AspireAI community · Hugging Face
AutoBench's Router Benchmark scored seven commercial and open-source routers on 252 long, open-ended prompts across ten domains, with twenty models answering each prompt and ten LLM judges grading every answer, tracking the real billed cost alongside the score. None of the seventeen headline settings occupied the region that would justify a router: above random quality and cheaper than a random pick. Always calling GPT-5.6 Luna reached 85.3 percent Delta-Hit at 0.0043 dollars per question, while always calling GPT-5.6 Sol scored only 87.5 percent at 0.079 dollars. Across 10,000 bootstrap resamples no setting beat that static policy on both axes, and across 59 cross-vendor router pairs the median rate of naming the same model was zero. Five of the seventeen settings send at least 90 percent of prompts to one model, so their quality score is just that model's score. The caveat: this is the benchmark authors' own study, and it was run against routers as shipped in late August and early September rather than against vendor-tuned configurations.
Read the story and explore more from the AspireAI community.
Discover more stories
Stay curious.
THE BIG PICTURE
A FRESH PERSPECTIVE
Research
One distance function matched sparse-autoencoder features across layers and compressed feature circuits into supernodes
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.
Swipe into something new
01 / 1729
The conversation continues
Explore Builds — projects by Kiran Adimatyam ↗
A fresh perspective changes everything.
ASPIREAI — STAY CURIOUS.