Trajectory-Aware Policy Optimization closed a state-distribution gap in long-horizon LLM RL and beat GRPO by up to 48 percent on math and 12 percent on code
AspireAI community · arXiv (TAPO authors)
*Reinforcement learning has become the default way to improve LLM reasoning, but long-horizon autoregressive rollouts carry a fundamental limitation: local policy mismatch compounds into severe state distribution shift over a trajectory.* Existing LLM RL methods such as GRPO and GSPO control that mismatch at the token or sequence level and overlook the trajectory-induced shift, which shows up as training instability and eventual collapse as the horizon grows.
Trajectory-Aware Policy Optimization derives a state-distribution correction from the off-policy policy gradient and folds it into the RL objective, adds a geometric-mean estimator for variance reduction, and applies a length-aware dynamic clipping that prevents overly deviated updates along the trajectory. Across eight competition-level benchmarks, TAPO significantly surpasses existing LLM RL algorithms, achieving up to 48 percent improvement over GRPO on mathematical reasoning and up to 12 percent on algorithmic coding, with substantially enhanced training stability. The paper frames the fix as closing a trajectory-level gap rather than another token-level trick, which makes it relevant to any long-horizon RL setup where rollouts compound drift.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.