A data-science agent that writes its own rubrics across attempts gained 24.5 leaderboard percentile points on the easy split and 14.8 on the hard one
AspireAI community · arXiv
Data-science agents can automate end-to-end machine-learning workflows but remain far from competitive on human leaderboards, failing through process errors rather than coding errors: insufficient exploration, unsound validation, and poor use of a fixed budget, such as re-running the same exploratory data analysis and baseline models on every attempt. AdaPRA is a persistent coding-agent harness that accumulates knowledge across attempts on the same task, with rubric-guided process supervision that identifies and corrects those errors. Its process reward agent audits the agent's trajectory and workspace by executing checks, then adapts its rubrics to each task.
Rubrics seeded from easier tasks generalise to harder ones, and rubric adaptation improves performance further. With five attempts per task, AdaPRA improves an 80B Qwen model's leaderboard percentile by 24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard while reducing process errors. The supervision also transfers across models - rubrics mined from one base model's runs improve another, DeepSeek-V3.2 - and AdaPRA beats DataPRM, a competitive process reward model for data science. Code is released for reproducibility.
The honest limit: the result depends on five attempts per task, so the comparison is against a single-attempt baseline and the gain is not free. The headline numbers are on an 80B Qwen backbone rather than a frontier closed model.
https://arxiv.org/abs/2610.33245
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.