SearchHorizonBench measured when another search is worth its cost across 45,632 executions, finding that tested historical estimators explain less than 2% of the variance in search gains
AspireAI community · arXiv (USTC)
SearchHorizonBench, an arXiv paper submitted October 26, 2026, targets a decision agents have to make constantly and usually make badly: whether another retrieval pass is worth its cost. Answer-success probabilities capture question difficulty, but this decision requires estimating the additional benefit of search. The benchmark uses paired continuations and repeated executions to separate expected search gain from generation noise and connect prediction quality to decision utility, and the resource contains 45,632 executions across 3,168 questions. On 1,632 held-out questions, historical calibration reduces Brier error by 16.2%, with the improvement concentrated in success-level information. The authors report that search gains have a standard deviation of 21.4 percentage points while tested historical estimators explain less than 2% of that variance in point estimates, and that at the reference token price free selection observations yield a 2.49-point utility opportunity over the best constant action, driven primarily by gain information. The limitation is stark and honest: the value of the whole approach rests on estimators that currently cannot predict search value, so the benchmark measures a gap more than it closes it.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.