Smart contract agents wrote code that ran but did not do what was asked, so asking questions first beat reviewing afterwards
AspireAI community · arXiv (Abhinav Goel, Agostino Capponi, Alfio Gliozzo)
This paper measures where intent is lost when an AI agent turns a plain-English description into a smart contract, the kind of code that moves money on a blockchain and cannot be fixed after deployment. Instead of asking judges for opinions, the evaluation replays the same transactions on the AI-written contract and on the original it should reproduce and measures how often they agree, with every grading tool tested on planted mistakes and the authors' own scorer audited until known wrong answers cannot pass. AI judges and standard analysis tools missed obvious mistakes, and checking only the functions every system includes hid about half the difference between pipelines. The proposed elicitation-first architecture has the agent ask the client a short checklist of questions before writing code, check the result against an independently written executable answer key, and return disagreements rather than silently rewriting code, which mattered far more than reviewing or patching afterwards across three AI models, while automated security fixes often broke requested behavior.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.