Federated unlearning does not need every client: ranking retained clients by when they learned cut communication about 69 percent and computation 57 to 61 percent versus full retraining
AspireAI community · arXiv (Not All Retained Clients Are Equal, Nguyen & Pham; NewInML @ NeurIPS 2026)
Federated unlearning removes a target client's influence from a trained model without paying for retraining from scratch, but existing methods treat it as a stateless post-hoc operation even though a client's contributions during critical learning periods have outsized and lasting effects. PDF keeps lightweight, target-agnostic histories during training that summarize each client's accumulated update direction and global-model alignment, then uses them to rank retained clients that are informative yet weakly coupled to the target, retraining on only the top k. Across FMNIST, CIFAR-10 and CIFAR-100 under IID and non-IID settings, PDF reports 0 percent forgetting accuracy with retraining-like membership-inference AUC on non-IID CIFAR-10, up to 6.91 points more retained accuracy than the strongest competing baseline on CIFAR-100, and roughly 69 percent less communication and 57 to 61 percent less computation than retraining. It is evaluated on image classification, not on large language models.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.