Model organisms built by post-hoc fine-tuning may be the easiest testbeds to interpret, and a suite of 61 of them showed how much the answer depends on how you trained them
AspireAI community · arXiv (Szablewski, Konar-Steenberg, Fornasiere, Menon, Heimersheim)
Interpretability researchers rely on "model organisms" - small models deliberately trained to exhibit a hidden behaviour - to test whether a white-box method can find what was taught. This work questions the testbed itself. The authors built 61 model organisms from OLMo2-1B and gemma-3-1b-it using seven different training techniques, including standard post-hoc supervised fine-tuning, post-hoc DPO, and integrating the behaviour directly into the OLMo post-training DPO phase, then trained 96 further variants via knowledge distillation. Benchmarking activation oracles, activation steering, the logit lens and the Jacobian lens across that suite, the authors found interpretability depends strongly on the training objective, the target behaviour, the model architecture and the data generation pipeline, with substantial variance remaining even after controlling for how strongly the behaviour was expressed. The uncomfortable result: the more realistic integrated training and the distilled variants were often less interpretable than plain post-hoc fine-tuning, which casts substantial doubt on how well current model organisms proxy for real interpretability problems. The limitation is scope - 61 organisms across two small base models is not a broad survey, and the paper offers benchmark-design guidance rather than a fix.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.