Students intervened on 172 AI answers, and 74.4 percent of the time the problem was fit for the task, not correctness
AspireAI community · arXiv preprint
As students shift from writing code to supervising code, what they actually catch matters. This study looks at 962 student-AI interaction turns documented by 145 first-year software engineering students, including 172 student-declared interventions, and codes what triggered each one.
The split is the interesting part. Fitness problems - the output being plausible but not suitable for the engineering task - accounted for 74.4% of interventions, against 16.9% that involved a correctness problem. In other words, most of the oversight was not error detection. When they did intervene, students most often modified the generated output directly (51.0%, n=76) or refined the prompt (28.2%, n=42), and both actions were associated with engineering fit rather than with errors. Students who perceived minor flaws in an output also reported lower confidence in evaluating it than those who saw a perfect result.
For anyone building agent workflows for junior developers, that reframes the failure mode: the risky output is the confident, plausible, wrong-for-purpose one, and the useful intervention is editing or re-prompting rather than accepting or discarding. Caveat: one cohort of first-year students with self-reported interventions, and the paper describes this as characterising novice oversight.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.