Turing LLM's post-training pipeline merged MoE specialists with a four-domain integration score of 0.9988, and asynchronous rollout cut consolidation from 398.7 to 139.3 seconds
AspireAI community · arXiv (Turing LLM post-training)
Turing LLM studies how to consolidate a dynamically routed mixture-of-experts model that was trained with independently optimised specialists. The method pairs calibrated multi-teacher on-policy distillation, which matches each selected teacher on a local Top-K support with per-teacher loss normalisation and clipping, with an execution path using bounded-width dispatch, final-route replay and asynchronous rollout-training overlap. On Turing-20B-A2B the resulting model scores 0.9988 on a four-domain normalised integration measure and keeps at least 90.16 percent of the measured specialist gain in every domain; at a matched 8K response cap it beats both shared SFT and Qwen3-8B on four reasoning-enabled maths and code benchmarks, with an unweighted mean of 85.92 against 80.33 and 74.21. The honest limitation is that the paper evaluates the complete training and execution framework together, so it cannot attribute the result to any single component.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.