Scoring claims against their own evidence across 7 systems and 4 datasets found research agents consistently assert conclusions stronger than their citations support
Deep research, meaning using large language models to read a large document collection and produce insights, requires synthesizing information across different sources. The paper finds that LLMs produce long-form research reports that read as confident syntheses but which fail to accurately reflect the underlying documents, a failure the authors call smoothing: in collapsing diverse and potentially conflicting sources into a single narrative, the model discards the evidential jaggedness of its corpus and asserts conclusions more decisive than the citations can support. SmoothScore is a claim-level scoring pipeline that extracts key conclusive claims from model-generated reports, finds evidence relevant to each claim and clusters that evidence along high-level aspects, then scores each claim with respect to the evidence to quantify the degree of smoothing. Across 7 systems and 4 datasets, agents consistently produced stronger claims than warranted. The limitation: the pipeline's own claim extraction and evidence clustering are model-driven, so the measured degree of smoothing depends on how well those components work, and the abstract reports the direction of the effect without a single headline magnitude.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.