RefactorProof rebuilt 185 Lean tasks into 458 behaviour-preserving variants, and most proofs stopped applying the moment the code was refactored
AspireAI community · arXiv
RefactorProof asks whether a verified proof survives a refactor that preserves program behaviour, by deriving 458 Lean-certified variants from the 185 VERINA tasks that compile in the authors' pinned environment while holding the specification and the evaluated proof fixed byte for byte. The answer is largely no: task-normalized structural proof survival is 4.3 percent for the supplied reference proofs and 13.3 to 26.8 percent for specialised provers, though the specialised provers still beat the references by 15.6 to 20.3 points on matched tasks with confidence intervals excluding zero. The comparison narrows to 21 to 25 short non-recursive tasks, and the paper is explicit that it measures unchanged-proof stability rather than the cost of repairing a proof that breaks, which is much lower.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.