Research-ideation agents scored against ICLR, NeurIPS and ICML reviewer comments all score poorly on scenario novelty and rarely change the problem setting
AspireAI community · arXiv
Research ideas are always proposed under resource constraints - a plan must fit the available compute, API credit and time - but evaluations of research-ideation agents mostly score unconstrained novelty or overlap with a published paper, and their LLM judges are rarely checked against peer review. AlphaIdeaBench is a benchmark of 100 ideation tasks (50 workshop calls for papers, 50 open problems stated in surveys), each paired with an explicit resource budget and a shared proposal schema, scored by an LLM judge on 12 dimensions covering validity, novelty and feasibility.
To validate the judge, the authors compare its scores on four novelty axes - problem, method, scenario and insight - with reviewer-derived scores from 600 papers across ICLR 2026, NeurIPS 2025 and ICML 2025. On problem and method novelty, Gemini 3.7 Flash is within one point of the reviewer-derived score on 94 to 97 percent of papers, with lower agreement on scenario novelty. Eight published harnesses were evaluated on a shared backbone and, within one harness, eight backbone LLMs: CoI-Agent scores highest overall, narrowly ahead of SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely change the problem setting. The backbone largely determines which open problem a proposal targets; the harness determines whether a proposal addresses one problem or combines several.
The honest limit: a judge validated against reviewer comments on three conference years still inherits the limits of those conferences' review practices, and weaker backbones such as Llama 4 Scout fail to carry out the harness strategy at all. Tasks, rubrics and scores are released.
https://arxiv.org/abs/2610.33644
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.