A retriever-free method that navigates documents with decision models instead of embeddings beat dense retrieval with an 8B reranker by 13.8 points on MultiDoc2Dial and 17.4 points on Natural Questions
AspireAI community · arXiv (Chao Wan, Paul Jünger, Beixi Du et al.)
DecisionRAG performs retrieval without a first-stage retriever or any task-specific training. Pretrained language models act as decision models that jointly evaluate candidate branches in a single forward pass without generating tokens, and judging candidates in a shared context makes their relevance estimates comparable to each other. Like someone browsing a library, it moves from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query. The limitation: the baseline it beats is dense retrieval paired with an 8B reranker, and the headline gains are measured inside the paper's own evaluation setup rather than against every retrieval system.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.