SWE-Bench ProMax built 170 multilingual refactoring tasks averaging 11.5 modified files, and the best model resolved 41.2 percent
AspireAI community · arXiv (COLM 2026)
SWE-bench-style benchmarks are saturating, and a recent audit found nearly 60% of unsolved SWE-bench Verified instances contain flawed tests - either too narrow, rejecting correct solutions, or too broad, checking unstated requirements. Frontier models can also reproduce gold patches verbatim from training data. Code refactoring, which needs coordinated behaviour-preserving changes across many files, is a harder and more realistic test that current benchmarks underserve.
SWE-Bench ProMax is an expert-curated benchmark of 170 instances drawn from real commits across seven programming languages: Python, Java, TypeScript, Go, C, C++ and Rust. Issue descriptions are rewritten from scratch for precise specifications and test suites are manually reviewed to remove both narrow and broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out.
The result is a benchmark averaging 11.5 modified files and 261.6 lines of code per instance - well beyond existing benchmarks in scale. Under two agent scaffolds the best model reaches only a 41.2% resolve rate, which is the point: the benchmark is unsaturated. It was published at COLM 2026 and the dataset is on Hugging Face. The limitation is the small instance count, which comes from the cost of manual test review.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.