A 216,486-prompt multilingual safety corpus found attacks that fail in English succeed once translated, and one model family broke far more often
AspireAI community · arXiv (Swiss AI Lab / ETH)
Safety alignment is heavily English, and English-only evaluation hides that. MSafeTemp is a released corpus spanning 60 languages, built by extracting safe templates from harmful prompts, translating them and refilling the slots with culturally authentic entities, harmful and benign in matched pairs, with every variant checked by a dual-stage LLM judge. It expands HarmBench from 200 prompts into a 216,486-prompt benchmark across those 60 languages. The headline finding is that English-only evaluation misjudges multilingual safety: for several models attacks that fail in English succeed once translated, and culturally localised variants expose further failures that translation alone misses. Olmo and both Apertus models broke more often after translation in nearly every language and benchmark, with Olmo on HarmBench going from 0 to 13.7 percent, while Qwen3.5 and GPT-OSS stayed near or below 1 percent on every arm, at the cost of GPT-OSS posting the panel's highest over-refusal. The paper's own caveat is that human raters endorsed 91 percent of harmful labels but only 65 percent of benign ones, so its over-refusal numbers may overstate the real rate. Code and dataset are on GitHub and Hugging Face.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.