Only 56.7 percent of Google's cited AI Overview statements were actually supported by the page they pointed to, and 22.2 percent of health answers made the claim physicians wanted avoided
AspireAI community · arXiv (University of Michigan / health informatics)
A study of 3,611 benchmark health questions compared Google's AI Overviews against the traditional top-five search results for the same queries, on source selection, credibility and factual grounding. Overviews appeared for 96.4 percent of questions, but they drew on substantially different evidence: only 44.4 percent of the traditional top five results were cited at all, and AI Overview citations over-represented commercial health publishers such as WebMD and Healthline at the expense of academic and government sources, scoring lower on four of six established credibility criteria. Grounding was weak too - after correcting for classifier error, an estimated 56.7 percent of cited statements were actually supported by the page they pointed to, and a further 20.8 percent of all statements were factual claims with no citation whatsoever. Graded against physicians' rubrics the Overviews earned 38.2 percent of available points, with 54.2 percent on accuracy and 21.6 percent on completeness. The sharpest finding is that on the 2,065 questions where a physician named a specific inaccurate or potentially harmful claim to avoid, 22.2 percent of Overviews still made that claim, including 28 percent of the ones that had passed a general factuality check. One encouraging number: emergency-referral questions got appropriate urgency 94.9 percent of the time. The limitation is scope - the evaluation covers Google's own health-ranking surface with English questions and physician-written rubrics, and does not establish whether grounding or accuracy improved since data collection.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.