Web agents degrade systematically when the environment shifts under them, and execution shifts - tools changing feasibility - hit hardest
AspireAI community · arXiv
Skill learning lets web agents reuse prior experience for long-horizon interactive tasks, but it is usually evaluated in closed environments, which overlooks how real web deployment shifts the invocation context, the perceptual grounding, and whether a skill is even still executable. SkillShift-Bench is a benchmark for evaluating skill learning under controlled open-environment shifts, simulating three kinds of shift - Contextual, Perceptual and Execution - across three self-hosted web systems, with diagnostics covering task success, interaction cost, skill use and skill failure.
The experiments show systematic robustness degradation under controlled shifts: lower success rates, higher interaction costs and more frequent skill failures, with Execution Shift exerting the strongest pressure. The authors go further and formalise shifted skill execution as a nonstationary local executability problem, giving a dynamic-regret analysis and a practical test-time adaptation method that improves learned-skill execution without relearning the skill library. Code, tasks, environments, documentation and a leaderboard are released.
The honest limit: the benchmark runs against three self-hosted web systems the authors built, so the shift conditions are controlled and reproducible rather than the messier churn of the live web. The adaptation method is shown feasible, not deployed. Accepted to EMNLP 2026.
https://arxiv.org/abs/2610.33528
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.