Pruning 30 percent of a model's neurons inverted its internal circuit while perplexity said it was barely damaged, and pruning heads destroyed the circuit outright
AspireAI community · arXiv
*Compressed language models are judged almost entirely on accuracy and perplexity, which quietly assumes that preserved output behaviour means preserved internal computation - and a controlled GPT-2 study shows that assumption fails in both directions.* Crossing two importance criteria, two component types and three sparsity levels, and measuring the fully characterised indirect object identification circuit with activation patching, path patching and direct logit attribution, the authors find two distinct failure modes: magnitude pruning of attention heads zeroes the circuit, output-facing heads are removed and the model falls to near chance, while magnitude pruning of MLP neurons at 30 percent leaves all 26 circuit heads intact but flips the sign of the Name Mover contributions, so the model systematically prefers the wrong name at 17.7 percent accuracy. The sting is that perplexity ranks the inverted model as less damaged than random head pruning, 74.6 against 140.1, and a replication on the Greater-Than circuit shows the critical component type depends on the circuit's own architecture. The practical implication is that output metrics and structural checks can each miss mechanistic damage, which argues for circuit-level causal analysis alongside them when evaluating a compressed model. The limitation is the obvious one, that everything is measured on GPT-2, so the specific thresholds should not be read as guidance for production pruning.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.