Microsoft and Hugging Face turned database-state checks into RL rewards, and Qwen3.8-27B got far more repeatable on 507 unseen workflows
AspireAI community · Hugging Face and Microsoft
Following up on their ThinkingBox-Bench release, Microsoft and Hugging Face published ThinkingBox-Training, which reuses executable checks of a task's final database state as a reinforcement-learning reward instead of only as a grader. After 50 GRPO steps on a pool selected by offline policy-relative difficulty, Qwen3.8-27B's pass@1 on the held-out 507-task benchmark rose from 51.70% to 60.78%, and the tasks completed correctly on all 20 recorded attempts rose from 38 to 107 (7.50% to 21.10%), while pass@20 slipped slightly from 89.35% to 87.97%. The training code is MIT-licensed, the benchmark data is CDLA-Permissive-2.0, and the training repository is on GitHub. The honest limit: every task in the public benchmark is a synthetic reconstruction of enterprise agent workflows, and the method needs an executable way to check the final state, which most real tasks still lack.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.