Scoring a world model on how well its predictions match the future rewards copying, and ScratchWorld replays Scratch projects to measure what models actually compute
AspireAI community · arXiv
World model evaluations usually score a predicted future state by its overlap with ground truth, which in an interactive world rewards a model for preserving persistent state rather than for predicting the consequence of an action. ScratchWorld treats Scratch projects as executable worlds and derives every label from replay on a pinned Scratch VM, covering next-state transitions, hidden variables, causal traces and counterfactual outcomes, with a primary metric that only credits a prediction when it names a changed field and computes its executed value. The anchors make the confound explicit: a no-change prior scores 91.4 percent on full-state field accuracy for synthetic long-horizon items but 0 percent changed-field credit on non-empty one-step items, while VM replay reaches 100 percent wherever the metric is defined. Across seven frontier LLMs prompted as world models, state-only scores reach at most 30.0 percent on controlled synthetic mechanisms and fall steeply as projects grow more complex, and even revealing the full program only lifts the best model to 55.4 percent, with causal-event accuracy staying near the 50 percent no-event baseline. The authors' summary is that ScratchWorld separates knowing the rules from executing them, and the limitation is that Scratch is a deliberately simple world.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.