Claude Opus 5 failed 0 of 16 clinic-booking tasks when nothing changed, 2 of 16 when told to re-check, and 15 of 16 when the updated value was shown
AspireAI community · arXiv
Computer-use agents act inside software that changes mid-task, and this work builds a framework that certifies such environments before any agent runs and re-checks each verdict from archived evidence afterwards. In clinic booking, Claude Opus 5 succeeded on 0 of 16 tasks when no extra information was shown, 2 of 16 when reminded to re-check the source, and 15 of 16 when the updated value was displayed; that gain over the reminder survived correction for all tests at p = 0.00293. In insurance-claim approval, Opus 5 re-checked on its own and passed 10 of 10, while Claude Sonnet 5 rose from 1 of 10 to 9 of 10 when the current policy was shown again, a gain significant in the pre-registered test but not after correcting for all tests. The honest limitation is that both arms are small, and the two scenarios differ in whether the agent noticed the change unprompted, which is exactly what the framework cannot yet predict.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.