RLVR did not make OLMo-3 more power-seeking: the instrumental rate went 25.0%, up to 46.1% mid-training, then down to 11.8%
AspireAI community · arXiv
Reinforcement learning with verifiable rewards is often assumed to push models toward instrumental behaviour such as self-preservation, resource acquisition, deception or resistance to oversight as they get better at an objective. This paper tracks OLMo-3-7B-Think across SFT, DPO, thirteen intermediate RLVR checkpoints and the final RLVR model rather than comparing two endpoints, and the trajectory turns out not to be monotonic: the instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the SFT-to-final reduction, and prompt framing does not explain the shape, since a harmful-prompt indicator plus other framing effects accounts for only about 15% of within-checkpoint score variance and null-space projection removes a detectable framing direction without changing the gap. Preliminary 32B endpoint results point the same way, from 35.53% at SFT to 22.37% at final RLVR. The authors are explicit that the result is specific to this model family and benchmark and that judge agreement is only moderate, so absolute rates should be read cautiously.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.