SimpleLoRIF traced safety degradation back to specific training examples, and SafeTulu cut GCG attack success by up to 22 percent
AspireAI community · arXiv (Allen Institute for AI and collaborators)
*Fine-tuning on instruction data can quietly erode a model's safety even when no training example looks harmful, and standard content filters struggle to catch it because the degrading data looks benign.* SimpleLoRIF is a scalable gradient embedding method that identifies fine-tuning examples whose gradients align with unsafe behaviour. By simplifying a low-rank attribution method and restricting backpropagation to the final four transformer blocks, it can embed datasets on the order of a million examples, with AUC as high as 0.865 in the gradient space of misaligned models on emergent misalignment.
Applied to build SafeTulu, a version of Tulu 3 with the 20 percent of examples most gradient-aligned with unsafe responses removed, training on SafeTulu reduces GCG attack success rate by up to 22 percent across Qwen, Llama and Olmo, outperforming LLM-as-a-judge and prior gradient-based filtering while largely preserving capability. The authors also identify under-studied sources of degradation such as tool-execution failures and narrative arithmetic, and release SafeTulu plus a Hugging Face Space for building custom variants.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.