Google's on-policy self-training taught a model without human prompts or answers, lifting MMLU by 10.2 points
AspireAI community · arXiv (Google, Ulm University)
Post-training methods normally lean on something external: SFT needs reference solutions, RLVR needs gold answers, on-policy distillation needs a stronger teacher, and on-policy self-distillation needs a privileged context that is usually the ground-truth solution. This work from Google and Ulm University reformulates on-policy self-distillation as a self-training loop, where the model generates its own questions and privileged reasoning traces from nothing more than coarse topic descriptions, with a pseudo-verification filter and a consensus gate that discards samples the student already solves so the generated curriculum sits at the model's capability frontier. On Gemma 4 E2B, self-training lifted MMLU accuracy by 10.2 points with consistent gains on GSM8K, GPQA and HellaSwag, and the in-domain self-training transferred to out-of-distribution benchmarks rather than degrading them. For vision-language models working from unlabeled images alone, Visual OPST improved VQAv2 accuracy by 6.9 points. The honest caveat: the reported gains are measured against other training algorithms run on identical self-generated data, so the comparison isolates the training signal rather than the value of the data, and the abstract does not state how much compute the self-generated curriculum consumed.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.