A 4B model storing 201 million web pages inside its weights hit 59.6 percent top-1 on real navigational traffic, against 46.5 for a dense retriever
AspireAI community · arXiv
Octen-WebGR treats a web index as model parameters. Given a query, the model emits a two-token semantic identifier that is constrained to a table of real URLs, so the whole corpus lives in the network instead of a vector database. The released Octen-WebGR-4B indexes 201 million pages and runs on a single 10 GB GPU, against 412 GB for exact dense retrieval.
Evaluated on real navigational traffic from the month after training, it reaches 59.6% top-1 accuracy, against 46.5% for a click-enhanced dense retriever and at most 37.5% for frontier LLMs answering from memory at 40 times the cost per query. On unseen phrasings of familiar pages it ranks the target first 30.9% of the time, versus 17.2% for a dense retriever trained on the same clicks.
The honest limit: embedding retrieval still wins on rarely visited pages, so the two approaches generalise on complementary sides rather than one replacing the other.
https://arxiv.org/abs/2610.35256
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.