52,168 agent trajectories: agent benchmarks score the model and its harness together, and the rankings flip between them
AspireAI community · arXiv (Shao Zhang et al.)
An LLM agent is a model plus the harness that runs it, so a benchmark number describes the pair rather than the model alone. This paper argues harnesses should be evaluated through their interaction with underlying models, and proposes five measures to do it: Valid Completion Rate, how reliably agents finish normally and deliver evaluable results regardless of task correctness; Model-Harness Fit, how well a harness supports task achievement for a given model; Cross-Model Fit, a harness's suitability across evaluated models; Interaction Residuals, pairing-specific strengths hidden by component-level summaries; and Token Load, the token cost of execution including failed attempts. The evaluation covers four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI and DeepAgents) with four core models plus two fixed native model-harness systems, over six benchmarks spanning context management, tool use, long-context planning, multimodal interaction and artifact handoff. Including native references and stress conditions, 111 evaluation settings produced 52,168 trajectories. Rankings varied across models and benchmarks, and task performance and valid completion sometimes favored different pairings; OpenCode ranked first on cross-model performance, while DeepSeek V4 Flash Preview held consistently high Mean VCR across harnesses. The honest caveat: the authors label it a preprint under review, and the conclusions cover the specific four harnesses and four models evaluated, so the specific winner should not be read as a general ranking.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.