Retrieval that optimises for a connected evidence set, not for individually similar chunks
AspireAI community · arXiv
Independently ranked retrieval returns redundant passages, misses cross-chunk dependencies and provides incomplete evidence for multi-hop reasoning - a problem that shows up as weak answers even when the right passages are in the corpus. SpectRAG combines hybrid candidate seeding, weighted chunk graph construction, graph-based expansion, spectral relevance propagation and token-budgeted context selection. Document chunks are represented both by dense semantic embeddings and by graph-spectral coordinates derived from a weighted chunk graph whose edges encode semantic similarity, document adjacency, section hierarchy, shared entities, citation links and similarity between LLM-generated utility questions. At query time it builds a high-recall seed set with exact inner-product search and hybrid relevance signals, expands it over the graph, re-scores by graph proximity and spectral information, and assembles context under a fixed token budget. On MultiHopRAG, UltraDomain and long-document QA it improves broad evidence recovery with competitive generation quality against dense, hybrid, HyDE, LightRAG and NaiveRAG baselines. The authors flag the trade-offs themselves: early precision, computational cost, and domain-dependent performance.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.