MuonGS dropped the weight-norm constraint and still got Omega(sqrt(T)) learning-rate accumulation, worth an estimated 15 percent token-equivalent speedup
AspireAI community · arXiv
*Weight decay and explicit norm constraints both help Muon pretraining, but the reason why and how much is needed were both unresolved.* This work analyses norm control across the whole training trajectory using accumulated effective learning rate, the running sum of the nominal learning rate divided by the weight norm. The analysis shows vanilla Muon can accumulate only logarithmically, which may limit exploration and help explain its slower empirical progress, while MuonW and MuonH guarantee at least linear accumulation under the stated conditions. MuonGS goes differently: instead of fixing the weight norm, it uses Gram-Schmidt projection to remove the radial component of each Muon update, which with bounded update directions and a constant nominal learning rate bounds weight norm growth to grow at most proportionally to the square root of the step count. That guarantees Omega(sqrt(T)) accumulation with sustained progress even late in training, plus an O(T^-1/2) stationarity guarantee for smooth scale-invariant objectives. In pretraining on 160M- and 400M-parameter dense models and a 135M-active/1B-total MoE, MuonGS reaches lower validation perplexity than MuonH at long horizons - an estimated token-equivalent speedup of up to about 15 percent at matched validation loss. The limitation: the scale tested is small, and the speedup is estimated rather than measured end to end.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.