Two models trained on identical data with identical safety training broke on different jailbreaks, and the best attack template survived a new pretraining seed in only 27 percent of pairs
AspireAI community · arXiv
Jailbreak benchmarks usually treat the set of attacks that break a checkpoint as a property of its data and safety training. This paper asks the harder question: would another pretraining run of the same data, given the same safety post-training, break on the same attacks? The authors build 150 models from 25 pretraining data recipes, three pretraining seeds and two post-training seeds, then collect 20 responses per model to a shared jailbreak prompt set and correlate per-prompt non-refusal rates.
Two post-trained copies of the same pretraining run correlate at 0.984. Changing the pretraining seed drops that to 0.811, while changing the data recipe lowers it only slightly further to 0.779. Practically, the best of 91 attack templates found on one checkpoint stays best after a new pretraining seed in only 27% of model pairs, against 67% after a new post-training seed. Each pretraining run also leaves a fingerprint: across 3,211 prompts a model's per-prompt non-refusal rates identify its run among all 75 every time, and a random 50-prompt subset still does so 98% of the time.
The honest limit: these are 1B-parameter models trained from scratch in a research setting, not production frontier checkpoints.
https://arxiv.org/abs/2610.34074
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.