A contrastive-style rubric judge hit 75.4% accuracy and stayed at 28 ms per judgment as rubric descriptions grew to 3,000 tokens
AspireAI community · arXiv
Rubric-based evaluation usually means prompting a large language model as a judge, which needs a generative call per judgment and reads every level description - so its cost grows with the rubric. RubriX borrows from contrastive language models instead: a LoRA-adapted encoder embeds the input and each level description, and a softmax over scaled cosine similarities produces a probability for every level without generating text.
Trained on public rubric data plus proprietary synthetic data, RubriX reaches 75.4% exact accuracy on held-out rubrics against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge, and beats G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets. With level embeddings cached its latency stays at about 28 ms per judgment as descriptions grow from 10 to 3,000 tokens, while Jev's rises from 44 ms to 211 ms and the Qwen3-8B judge's from 32 ms to 516 ms - making RubriX 7.7x faster than Jev and 18.8x faster than the Qwen3-8B judge at 3,000 tokens per level.
The honest limit: Jev keeps a larger lead on two of eight human-labelled benchmark sets and ranks responses best on average, so the speed win is the clearer result.
https://arxiv.org/abs/2610.34875
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.