An agent wrote the mathematical formulation, a solver searched inside it, and the result ran Qwen3-8B layers 1.075 times faster than SGLang on NVIDIA GB10
AspireAI community · arXiv
Designing AI hardware is usually done in layers, and the layering quietly rules out the wins that need the chip and its software to change together. Formulaic opens that boundary to a language-model agent with a three-way split of labour: the agent writes the formulation, a mathematical solver optimises within it, and verification and measurement check its assumptions and results. Each step is a template - a family of programs with the conditions under which they are correct - and wall relaxation searches the chip by pricing the hardware limits that bind the software. On real hardware the templates the agent wrote run complete Qwen3-8B layers 1.075 times faster than SGLang on NVIDIA GB10. In a simulated Ventus GPU study, joint chip-and-software design cuts worst-scenario cycles for Qwen2.5-0.5B to 0.428 of the reference against 0.934 for software alone, and reusing proofs across a five-step GEMM refinement chain makes checking 2.51 times faster. The paper is explicit that these are separate studies rather than one end-to-end run, and it reports its own negative results - in 18 transfer evaluations, Qwen-selected chips took 3.07 to 5.60 times the baseline cycles of GPT-2 small and Pythia-160M, and one prefill case where the joint choice was slower than the hardware-only choice.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.