Across 1,600 attempts on 50 banking tasks, agents often reached the right decision without completing the work, double-charging fees, splitting totals wrong and paying while approval was pending
AspireAI community · arXiv
Bank-World is a benchmark of 50 banking tasks across two simulated institutions, testing that agents not only compute correctly but reconcile records, obtain permission, execute transactions and keep instructions and status reports consistent as circumstances change, through single-turn assignments and multi-turn interactions with simulated employees who supply information, approve actions and revise instructions. Grading combines executable checks of recorded actions and state with model judgements of task requirements, including whether valid alternative solutions satisfy them. Across 1,600 attempts by four models the authors find that correct decisions often fail to become completed work, with agents charging fees twice, distributing the correct total in incorrect amounts among recipients, and executing payments while approval was pending. The models also differ, with Sol leading single-turn completion, Opus showing the highest observed multi-turn completion rate and Grok succeeding across more multi-turn tasks but less consistently. The limitation is that task environments, executable graders and full trajectories are retained privately to reduce contamination, so third-party replication is not currently possible.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.