Replit's ViBench 2.0 gives coding agents a whole product to build, and the best model passes every test plan in only 65 percent of its builds
AspireAI community · Replit
Most coding benchmarks hand an agent a well-defined task inside an existing codebase. ViBench 2.0 starts from an empty project instead: the agent builds a whole product one feature at a time, modelled on things like Uber, GitHub or Google Docs, then has to fit each new feature onto code it wrote itself. The benchmark spans 17 products, 241 feature requests and 544 test plans, with eight products public on GitHub in the Harbor format and nine held out, and every result is reproducible from one command on an OpenHands-based harness. The headline numbers are sobering: GPT-6.1 Sol leads with a pass@1 of 65 percent, GPT-6 Astra reaches 60 percent and Claude Opus 5.5 manages 49 percent, and no open-weight model passes a full build. Nearly three-quarters of the top models' failed builds fail only where features intersect - the test plans check how features work together, not just that each works alone. Cost is its own story: Sol runs about 12 dollars per build, Astra about 90, and DeepSeek V4.1 Flash passes 77 percent of its test plans for roughly 2.50 dollars - a useful price point, but not a complete build. Replit grades with Opus 5.5 after planting 34 bugs in working builds, and 99.2 percent of its passes were confirmed correct on re-grading.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.