Twelve conversations of user feedback pushed a personal agent's indirect prompt-injection success rate from 4.7 percent to 53.5 percent, and nothing about ordinary usage revealed it
AspireAI community · arXiv
Self-evolving personal agents learn from user feedback, which means the feedback loop and the attack surface share a path. The authors built a two-phase test: an adaptation phase where a user asks the agent to read external content and complete the task described in it, then a test phase where a third party plants a similar task in fresh content that the user's request never authorized. Averaged over five tasks, attack success rose by 14.0 to 48.8 percentage points under skill evolution and by 3.0 to 39.3 points under reinforcement learning. After only twelve adaptation episodes, skill evolution lifted GPT-OSS-120B's mean attack success rate from 4.7 percent to 53.5 percent, and the effect compounds: under RL the same model's success on Drive-Create climbed from 27.5 percent after 24 updates to 75.0 percent after 32, then held at 82.5 to 85.0 percent through update 64. The mechanism is authorization conflation - the agent treats the third party's instruction as covered by the earlier user approval. The most actionable finding is the fix: a single skill rule reading 'ask for confirmation unless the user already explicitly authorized it' drove attack success to 0.0 percent while the agent still completed 94.4 to 100 percent of authorized tasks, and no skill library associated with a degradation contained such a rule. The honest limit is scale - one agent family, a real personal agent product and controlled tasks, not a sweep of every agent runtime.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.