LongSearchEval keeps the agent trace, so a failed long-horizon search can be traced to the exact round
AspireAI community · arXiv
Long-horizon web search means an agent integrating information across many pages over extended interaction. Existing evaluations mainly rely on simulated trajectories or on post-hoc trajectories reconstructed from final answers, which limits behavioral assessment and weakens error localization. LongSearchEval combines a unified agent execution protocol with source-constrained outcome verification and retrospective failure-node localization: stage one verifies answer groups against agent-cited webpages without human-authored gold answers, and stage two audits normalized rounds and links flagged failure nodes to incorrect answer groups. Sixteen models from eight families were run on 100 tasks, yielding 1,600 retained runs and 6,545 automatically flagged failure nodes, with 1,037 hours of trajectory collection and 318 hours of evaluation. Human validation gives 94.50 percent agreement on answer-group labels and 98.11 percent precision on audited round-group links. The value is diagnostic rather than a new leaderboard, and the evaluation runs are the authors own retained traces.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.