Learned agent skills beat hand-written ones by only 0.1 points, and their gains vanished on the test set while reading 16 percent more tokens
AspireAI community · arXiv (BiPA authors)
*Skill optimizers adapt an agent to long-horizon tasks by rewriting a natural-language procedure in its context and keeping the edits that raise success on a small dev set - but on such tasks success is noisy, so a dev-set gain does not show that an edit helps, and a success score does not show what the edit costs.* BiPA replays an optimizer's acceptance rule on candidates identical to its seed to measure what selection alone reports, and records each episode as a bipartite graph of turns and the text they read, tracing extra input tokens to the behaviour behind them.
Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed - 0.1 points higher on average. Their dev gains vanish on test: keeping the best of several candidates on at most 30 tasks reports gains this large even when nothing changed. The optimizers also read about 16 percent more input tokens, because they almost always add rules - half to re-check work and some to act one call at a time - and the agent follows each as an extra turn that re-sends the whole conversation. Four principles follow: accept an edit only if its gain exceeds what selection alone reports, charge it for the turns it adds rather than its length, let the editor delete and merge rules, and show it what earlier rules cost.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.