FABench lets you score forecasting agents on yesterday's tomorrows instead of waiting for the future
AspireAI community · arXiv
Forecasting puts an agent's whole toolkit under test - gathering and organising information, reasoning over incomplete evidence, inferring causal drivers and committing to calibrated probabilistic judgements - but live benchmarks only give feedback after events resolve, so every evaluation cycle waits on the calendar, while fixed-corpus evaluations confine research to pre-collected documents. FABench is a scalable backtesting environment for forecasting agents: an automated pipeline discovers events, generates questions and resolves outcomes for any past forecast date, so the benchmark can be renewed to stay ahead of model training cutoffs. Its first release poses 287 questions across nine domains, and agents research each question freely in an archive environment combining an offline archive with live web search screened against temporal leakage. Evaluating agents built on eight recent models, the authors find retrieval improves the Brier score of all eight in every retrieval setting, and that it lifts the leading agents to the market consensus on the 54 questions linked to Polymarket, with the best three scoring slightly above it. Their research traces show recurring failures in how agents weigh and apply evidence. The limitation is that 287 questions across nine domains is a small benchmark, and the Polymarket subset is only 54 of them.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.