A blind benchmark of 108 real vulnerabilities found Snyk's static analyzer and an AI agent each missed about a fifth of them, and neither alone was enough
AspireAI community · arXiv
Whether AI security agents find what deterministic static analysis misses, and the reverse, is usually argued rather than measured. VulnBench 2.0 is a blind benchmark of 108 hand-curated, attacker-reachable vulnerabilities across 20 open-source web applications in Go, JavaScript/TypeScript, Python and Java, where a finding counts only if it locates both the attacker-controlled source and the dangerous sink within two lines. Snyk Code reached 82.4% macro recall and Claude Code with Opus 5 at xhigh effort 79.3%, and the two disagreed on 28 vulnerabilities; together they reached 101 of 108, the ceiling of all seven comparable configurations. The harness matters as much as the model - without Claude Code's built-in review policy, the same model matched 16 instead of 9 of the 20 vulnerabilities in restricted classes - and coverage correlated only loosely with cost.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.