A self-updating meta-analysis of 420 papers found human-AI teams beat humans but lost to the better partner
AspireAI community · arXiv (cs.HC, cs.AI, cs.CY, cs.DL)
The empirical literature on human-AI collaboration gives conflicting advice, and static meta-analyses cannot absorb new studies. AutoAnalysis is an automated, self-updating meta-analysis whose most recent run synthesised 1,955 effect sizes from 420 papers. The headline result is a distinction that reframes the field: human-AI combinations performed significantly better than humans alone, but significantly worse than the better of humans or AI alone. In other words, improving human performance is not the same as achieving human-AI synergy, and collaboration has to be evaluated against both baselines rather than against the human alone. The tool is designed to integrate new studies into the running meta-analysis as they appear. The authors frame it as cumulative evidence on when humans and AI perform better together, with results available on auto-analysis.dev.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.