A study of 194 agent runs found a 2.29-point apparent restart gain traced to a single failed attempt, and zero of 103 live comparisons met its own certification bar
AspireAI community · arXiv
If you are building a monitor that decides when to restart a stuck agent, the natural metric is classification accuracy, and this paper says that number can be almost uninformative. Working through an exact alarm-to-action identity, the authors show that a monitor with fixed 90 percent accuracy can produce anything from a five-point loss to a 35-point gain in success rate, depending only on whether the replacement attempt succeeds with the remaining budget.
They propose two replacements: a decision-sufficiency criterion that asks whether a progress representation preserves the information needed to pick between two specific actions, and a finite-sample decision audit that reuses existing agent traces with simultaneous binomial intervals to certify a preference for restart or continue, or to abstain when the evidence is not there. Applied to 194 executions on 19 SWE-bench Verified SymPy issues, the audit found a 2.29-point apparent restart gain that traced back to a single surviving failed attempt, a 42.29-point loss on a second model, no positive replication on prospective cohorts, and zero of 103 live comparisons clearing the certification bar.
The caveat is scale: 19 issues and 130 audited traces is small enough that single outcomes move the headline, and the certification threshold is the authors' own construction rather than an external standard. The underlying argument is the transferable part - measure monitors by the decision they enable, not by how well they classify.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.