A commercial System One model beat Qwen3.8-27B on 27 of 37 datasets and Gemma on all 37, for 346,009 requests costing under ten dollars
AspireAI community · arXiv
System One models do not generate text: given a state and typed questions they return a choice from fixed options, a position on a rubric, or a probability that a statement is true, which is exactly the shape needed for small decisions inside information-access pipelines. This evaluation scores Jev zero-shot on 37 datasets spanning classification, routing, moderation and rubric scoring, with one frozen template per dataset and full evaluation splits, totalling 346,009 requests for under ten dollars, and scores Qwen3.8-27B and Gemma-4-E4B on identical requests through their exact option probabilities. Jev reaches 95 to 99 percent on IMDB, SST-2, HellaSwag and ARC, beats Qwen on 27 of 37 datasets with none of Qwen's leads falling outside the bootstrap intervals, and beats Gemma on all 37, with well-calibrated choice probabilities that support selective prediction; extending the protocol to sequential decisions, Jev completes 35 percent of unseen ALFWorld tasks, 42 of 90 TextWorld games and 21 percent of BabyAI-Text missions, and picks the optimal Blackjack action in 90 percent of decisions. The honest limitations are real: all three models degrade on low-resource languages, noisy labels and rubric judgments, binary probabilities need tuned thresholds that lift micro-F1 on UNFAIR-ToS from 0.50 to 0.75, and the probes rule out shallow memorization but not memorized question-answer pairs.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.