Learning a reward over the path from a toxic state to a non-toxic one, then sampling backwards along it, cut toxicity by up to 69.38 percent at inference time with no model retraining
AspireAI community · arXiv
Large language models can generate toxic content, motivating effective test-time detoxification, while existing representation steering methods rely on static or coarse interventions and give imprecise guidance because they inadequately model transitions between toxic and non-toxic generations. Transition-Aware Reward-Guided Representation Sampling (TARGRS) is a test-time framework that learns reward guidance over toxicity transitions in latent representation space. During exploration, progressive steering along the toxicity direction constructs trajectories from non-toxic to toxic states, and the toxicity scores along those trajectories turn sparse annotations into dense, listwise supervision for a transition-aware token-level reward model. At inference TARGRS uses that model to guide coarse-to-fine sampling along the opposite, non-toxic direction: coarse sampling identifies the intervention needed to reach non-toxic regions, and refinement selects a detoxified representation with minimal perturbation. Experiments on 10 LLMs across RealToxicityPrompts, BOLD and AttaQ show TARGRS consistently outperforming state-of-the-art baselines, reducing toxicity by up to 69.38% on RealToxicityPrompts, while running nearly as fast as the original LLM and preserving core capabilities with minimal impact. Because it intervenes in the language decoder space it also extends to large vision-language models, reducing toxicity by up to 64.23% across three representative models. The limitation is that the approach requires an exploration phase to build trajectories for each model before it can be used, so the setup cost is not zero even though inference itself stays close to baseline speed.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.