A 325M model pretrained from scratch on an inference-only NPU matched a B200 GPU to within 0.0001 loss, and both checkpoints are public
AspireAI community · Hugging Face Blog — FINAL-Bench (SeaWolf-AI)
VKORE-325M-NPU-Base was pretrained from random initialisation on a commercial inference-only NPU, with every weight learned on that chip and backpropagation run on it, over 5.0 billion byte tokens of FineWeb-Edu in 76,292 steps across about four days. The authors ran the same experiment on a single B200 GPU with identical initialisation, data, batch and learning-rate schedule, so the hardware is the only variable. On 1,024 windows from a FineWeb-Edu shard that neither model saw, the NPU model reaches a loss of 0.7993 and the GPU model 0.7994, a paired difference of -0.0001 with a standard error of 0.0007, and the NPU model is lower on 50.1 percent of windows. Both weights and the evaluation script are published so the comparison can be checked directly. The engineering is the harder part of the story: the first long NPU runs trained correctly for thousands of steps and then diverged, a failure short tests did not reveal, and the authors say the stabilising technique was developed in house and is under patent application, so they describe the result rather than the method. Their own scope note is explicit: this is a 325M base model trained on 5B tokens, released as a training-hardware result and not as a capable assistant, and the 256 training-corpus windows they originally labelled held out are not strictly held out, which they corrected in the model card.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.