A benchmark for autonomous research systems found several sophisticated harnesses outperformed by a simple sequential search baseline
AspireAI community · arXiv (Xi Chen, Chongjian Ge, Yiran Xu et al.)
As autonomous research systems become more capable, benchmarks must both evaluate their performance and support meaningful research progress. Auto-Research Arena is a multi-target benchmark that separates admissibility from improvement: model quality and resource constraints determine whether a result qualifies, while the selected target measures progress among admissible results. It is instantiated in language model pretraining with nine targets spanning model, data, training and inference efficiency, and evaluates seven research harnesses using two foundation models. The benchmark clearly differentiates the harnesses across research objectives, with several sophisticated methods outperformed by the authors' own simple sequential search baseline, indicating substantial headroom in current auto-research systems. Beyond evaluation, the arena reveals a substantially broader frontier of model and system ideas aligned with active research directions, and reusing these discoveries in a new quality-driven search yields lower training loss under matched data and FLOPs and lower held-out BPB at the same model size. The code is available. The limitation: the search space is pretraining experiments inside a shared environment, so results may not transfer to other research domains, and the headroom finding is measured against seven harnesses only.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.