A membership inference attack on pay-walled books found GPT-4o scored 0.82 AUROC on content it was not licensed to train on, while older and smaller models sat near chance
AspireAI community · arXiv
Using a legally obtained dataset of 34 copyrighted O'Reilly Media books comprising 13,962 paragraphs, the authors apply the DE-COP membership inference attack to test whether OpenAI models show recognition of non-public, pay-walled content they were not licensed to train on, which goes beyond the usual training-inclusion question. GPT-4o exhibits patterns consistent with recognition of non-public books, with an AUROC of 0.82 and a 95% bootstrapped confidence interval of 0.60 to 0.96, where above 50% beats random chance. GPT-3.5 Turbo, trained two years prior, and the smaller GPT-4o mini both score near chance on non-public data at 0.54 and 0.56 respectively. The results are robust to independent paraphrasers (Gemma and Qwen) and to leave-one-book-out sensitivity analyses, and the authors additionally test GPT-4o and GPT-4o mini with the same October 2023 training cutoff to ensure the model is not simply identifying language shifts over time. The findings support disclosure requirements for pre-training data provenance and formal licensing markets. The authors state the results remain preliminary given the small book sample, possible contamination of post-cutoff controls, and differences across models in size, architecture and training composition, so the AUROC should be read as a signal rather than a settled measurement.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.