Turning the Turing Test around, GPT-5 interrogators read the source better than GPT-4 while humans found it harder
AspireAI community · arXiv preprint
The Turing Test treats a failure to tell machine from human as one undifferentiated result. This paper splits it into two: a failure of encoding, where the source information is simply not in the text, and a failure of readout, where the information is there but the interrogator cannot pull it out.
To separate them, the authors inverted the test and put humans and AI systems in the interrogator seat, then measured encoding, readout and meta-readout - the interrogator's explicit access to the features guiding its own judgments - across two experiment series with more than 3,800 human interrogators and two GPT generations. The result is a dissociation. As GPT-5 responses became more human-like than GPT-4's, source information became harder for human interrogators to extract, while GPT-5 interrogators read that information out more accurately than GPT-4 interrogators did. But the improvement came with a catch: GPT-5 explanations only partially aligned with the feature weightings that actually predicted its judgments, and the authors report a double dissociation between readout and meta-readout across human and GPT interrogators.
Read plainly: a model that is better at spotting the difference is not necessarily better at explaining why it did. Caveat: the paper has no HTML rendering and was read from the PDF text layer, and the findings are specific to the GPT-4 and GPT-5 generations studied.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.