LinMix treated pretraining data mixing as a regression design problem and beat Dirichlet sampling by 46 percent relative on MBPP
AspireAI community · arXiv
LinMix reframes the standard practice of picking pretraining data-mixture weights as a domain attribution problem, where the fitted regression coefficients measure how much each domain actually contributed to the target metric, and then fixes the sampling design that was making that attribution ill-posed. Instead of sampling proxy mixes from a Dirichlet distribution, LinMix samples at the vertices of the capped simplex so each domain is either at its cap or absent, which decorrelates domain effects and lets a linear fit work without hyperparameter tuning. The authors also prove that vertex sampling is optimal in this setting and that Dirichlet sampling inflates attribution error when mixes concentrate near the prior mixture. Pretraining 1B-parameter OLMo2 and 4B Qwen3.5 models, LinMix beat Dirichlet-based mixing across reasoning, code and math with relative gains of 46 percent on MBPP and 14 percent on GSM8K, and at 288 fine-grained domains it improved benchmark accuracy by 11 percent on average over the best nonlinear baseline. The gains come from proxy-model training runs at 1B and 4B scale rather than at frontier scale, and the venue is a NeurIPS workshop paper rather than a main-track result.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.