Tool retrievers that look fine on single-tool queries fall apart when an agent needs four at once
AspireAI community · arXiv
An agent that must call several tools fails outright if the retriever omits any one of them, yet tool retrievers are mostly built and evaluated on single-tool queries. This work compares six retrieval structures - flat lexical and dense retrieval, LLM query decomposition, LLM-navigated hierarchies, server-then-tool matching, dependency graphs and trained set-level retrievers - across two multi-tool testbeds: 345 AppWorld queries with expert dependency labels and 103 MCP-Bench tasks over 257 MCP tools, measuring completeness@k (whether the entire required tool set appears in the top k) alongside LLM calls, tokens and latency. On all 7,961 ToolRet queries the authors find completeness at 10 runs .58 to .67 on single-target queries but only .08 to .17 on queries with four or more targets, and that the ranking of retrievers on single-target queries reverses on multi-target ones. Crucially, none of the structures untrained on the testbed and making at most one LLM call per query completes more than 10% of AppWorld queries at k=16; a set-level retriever trained on 75-90 in-domain queries reaches .33 on AppWorld and .58 on MCP-Bench with no LLM calls, while an LLM-navigated hierarchy reaches .30 and .40 but needs a median of 31 to 38 LLM calls and about five minutes per query. In a downstream check with the official AppWorld agent, tasks with a complete retrieved set succeeded 21 of 21 times versus 2 of 39 for incomplete ones.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.