MLE-World, a world model finetuned from Qwen3.5-4B, predicted the terminal output of held-out ML training programs 77.9% of the time versus 41.1% for the best prompted model, making test-time scaling affordable for AI research agents
AspireAI community · arXiv (Meta / NYU)
MLE-World, an arXiv paper submitted October 27, 2026, addresses a practical bottleneck for AI research agents: every candidate program must actually be trained and evaluated before it can be judged. MLE-World is a world model finetuned from Qwen3.5-4B that predicts a program's terminal output - error traces, training logs and validation scores - from its code and data description without executing it, plus a confidence score estimating whether the prediction is right. On 17 MLE-bench Lite tasks the authors report that MLE-World's predicted terminal output matches the real run on 77.9% of held-out programs, compared with 41.1% for DeepSeek, the strongest prompted baseline, using 13 to 1,061 times less GPU inference time. Because a prediction costs one inference call rather than a training run, the authors built five test-time-scaling policies guided by the world model, which achieve up to 22.1% higher observed mean performance than the same agent without one while reducing real execution. The limitation the paper is explicit about: on held-out programs 22% of predicted terminal outputs and 14% of predicted exception classes disagree with the real run, and a visible SyntaxError makes the model predict every SyntaxError even though those are only 8.2% of training crashes, so the authors run a candidate for real whenever the confidence score is low.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.