Metis-1.6: two models built by a single researcher reached second of 30 in mathematics on a twelfth of the comparator training data, and the report is mostly about what broke
AspireAI community · arXiv
Metis-1.6 is two models trained from scratch by one researcher, Praxis at 2.94B parameters (0.46B active per pass) and Logos at 10.23B (1.18B active), using a hybrid Mamba-2 and attention mixture-of-experts stack run twice per token, with each trained on 1.02T tokens: 804.76B of pretraining, 18B of context extension to 131,072 tokens and 200B of supervised fine-tuning. The report is explicit that the models mostly do not challenge frontier models: Logos averages 52.93 across 40 post-trained benchmarks, seventh of 17 models in its comparison set and within a point of Qwen3-4B, and it is weak at code, instruction following and long context; its strength is mathematics, where it places second of 30 with 67.50% on AIME 2025 and 63.64% on HMMT February 2026, on roughly a twelfth of the median training data of the comparators that disclose it. The most valuable part is the failure log: an optimizer that never ran, attention trained with AdamW throughout pretraining, a 1.77B-parameter n-gram memory that learned to switch itself off, five costly fine-tuning choices, and a length reward that punished correct long solutions once a group's pass rate passed 2/3, which the author proves is a sign inversion and fixes with a one-line clamp.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.