Measured before trained: a 739-email inbox study found logistic regression beat the LLMs at routing
AspireAI community · arXiv (Federico Lucca)
Personal email agents decide what every message is, whether it is urgent, and whether to draft a reply - and their error rate is rarely measured, because the stored labels were written by the system itself. Foyer is a self-hosted inbox agent in daily use, paired with a measurement protocol that produces ground truth before any training: stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses and human labels in the review loop. On 739 real Italian and English emails (231 unbiased, 500 adversarially sampled), nine systems were compared: the production keyword rules scored 28% on message type against a 58% majority class, and with about 200 labels logistic regression had the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90) - statistically tied with Gemma 4 on type, with few-shot prompting not closing the gap. Two findings transfer beyond email: the sender's relationship is recoverable at 98.6% accuracy from other emails of the same contact but is not inferable from a single message by any model, and accuracy is not the product metric - the most accurate system most often ignored emails that needed action (18% versus 4% for the best LLM). The limitation is real: this is a single-user case study and the emails are private and not released.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.