Lifting entropy from the token level to the response level gave supervision-free credit assignment for multi-turn agent RL, adding 1.4 percent inside a state-of-the-art code RL framework
AspireAI community · arXiv
Multi-turn agentic reinforcement learning runs into a credit-assignment wall: sparse outcome-only rewards say little about which step inside a long trajectory deserved credit, and the usual fix, dense intermediate supervision from process reward models or self-supervised signals, adds tuning complexity and can hurt generalization. AEM instead modulates entropy dynamics adaptively during training, and its analysis argues that in agentic settings the environment responds to a complete response rather than an individual token, so uncertainty has to be measured at the response level to avoid chasing sampling noise. The method derives a response-level uncertainty proxy and rescales advantages with it, letting the balance of positive and negative samples shift naturally from exploration to exploitation. Experiments on ALFWorld, WebShop and SWE-bench-Verified with models from 1.5B to 32B showed consistent improvements over strong RL baselines, including a 1.4 percent gain when integrated into a state-of-the-art software-engineering RL training framework, and the authors also report that entropy drift under natural-gradient updates is governed by the interaction between a sampled response's advantage and its relative surprisal. The honest read is that the headline gain is modest and measured against a particular strong baseline.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.