On the same sub-frontier model, two independent coding harnesses scored 0.54 and 0.53 on a five-year VAT reconstruction while two vendor harnesses scored zero
AspireAI community · arXiv (GSequist)
*Most agentic benchmarks test generative work - build a model, a memo, a spreadsheet - while the expensive part of knowledge work is investigative: recovering a defended answer from incomplete, inconsistent evidence.* kw-bench is a benchmark built from real engagements. Its first task family is a five-year VAT reconstruction across three generations of SAP, delivered as raw system extracts with page headers, local number formats, partial link tables and no documentation of how anything connects. Grading is deterministic, with no LLM judge.
Four harnesses - Claude Code, Codex, OpenCode and micro-cc - ran on the same model, DeepSeek v4.1 Flash, for two hours. The two independent, model-agnostic harnesses delivered 0.542 and 0.528; the two lab harnesses, away from their home model, delivered 0.00 each. On a frontier model the task collapses: micro-cc and Claude Code on Claude Opus 5.5 scored 0.9986 and 1.000 in 22 and 25 minutes. The authors attribute the sub-frontier failures to inertia - keeping many weak signals alive across a long session - and note the practical implication that below the frontier, the harness decides how far a model gets.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.