Codex with DeepSeek-V4.1-Flash passed 82.8 percent of generated test suites but only 33.8 percent were behaviourally adequate at catching mutants
AspireAI community · arXiv (Req2Test-Bench authors)
*As agents turn natural-language requirements into repository-level code, requirement-to-test generation matters because tests give code generation something to validate against - but existing test-generation benchmarks focus on white-box testing, and semantic alignment between a document and a generated test is usually only judged by reading it.* Req2Test-Bench contains 5,086 execution-validated instances recovered from developer-authored behavior-driven development artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each pairs a Gherkin requirement with the developer's own Test Implementation, and the agent must produce a complete, potentially multi-file test.
Evaluation of 8 agent-model-reasoning configurations exposes a large gap between execution success and behavioural adequacy, measured strictly as kill-completeness: a test is adequate only if it kills every mutant the ground-truth test kills. The best configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8 percent of instances yet only 33.8 percent are behaviourally adequate. The authors find that scaling up agents yields sharply diminishing returns in adequacy relative to generation cost, that under-specified requirements cause the information gap through ambiguous setup and implicit expected outcomes, and that requirement-grounded harnesses steering agent non-determinism are a promising direction.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.