Ask a model to tell you everything it knows about a topic instead of answering a question, and 10,000 entities reveal what benchmarks were never testing
AspireAI community · arXiv (cs.CL)
BeQu introduces open knowledge evaluation, which scores what a model volunteers when given an open-ended elicitation prompt such as tell me everything you know about a person, rather than only the knowledge a benchmark designer chose to query. The benchmark pairs 10,000 entities with reference corpora for statement verification, and the authors analyse how reasoning effort, model scale, prompt format and knowledge domain change what comes out. The point is about coverage bias in existing benchmarks, not a new leaderboard headline: data and leaderboard are published.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.