A surgical video QA benchmark built to resist language shortcuts left every leading model below 53 percent, and expert surgeons reached only 64.9 percent
AspireAI community · arXiv
A capable surgical AI model should reason from visual evidence, yet many existing surgical benchmarks can be partially solved without visual input, as models exploit procedural priors or common anatomy-instrument interactions to infer answers from language alone. SUREON is a video question-answering dataset and benchmark designed to evaluate visually grounded surgical reasoning: the pipeline generates QA pairs from narrated surgical academic videos, refines incorrect options using multiple models to reduce linguistic biases, and uses trained adversarial guessers to filter out questions answerable without visual evidence. The resulting dataset holds 76K training pairs covering 168 procedures and 10 specialties, alongside a benchmark of 340 questions across 10 categories verified by expert surgeons. Benchmarking leading commercial and open-source models shows surgical visual reasoning remains largely unsolved, with no model exceeding 53% accuracy and expert surgeons reaching only 64.9%. The authors also introduce SUREON-VLM, a baseline trained on the curated data that avoids language shortcuts, outperforms existing open-source models on SUREON and matches larger specialist models on multiple standard surgical benchmarks. Training ablations show distractor refinement and blind curation strengthening visual grounding, increasing the multimodal gain from 8.7 to 29.1 percentage points while reducing the training set from 176.6K to 76K filtered samples. The limitation is the small 340-question benchmark, which is appropriate for expert verification but gives a wide confidence interval on the reported accuracies.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.