Chain-of-thought reliance follows a scaling law: answer reliance rises from 16-18% at 0.6B to 97-100% at 14B, tracking step accuracy at r = 0.99
AspireAI community · arXiv
Monitoring a model's chain of thought only helps safety if the final answer actually depends on the written steps, and prior measurements used natural tasks where true reasoning is unknown. This work measures reliance against exact ground truth on a synthetic box-swap task, replacing one step inside a model's own chain with a wrong but valid state and checking whether the answer follows the edit. Across 25 open-weight models on 7,200 items each, reliance rose from 16-18% at 0.6B to 97-100% at 14B in Qwen3, fitting a logistic in log N with R^2 = 0.89, and across all models reliance and step accuracy correlated at r = 0.99, with equally capable models equally reliant regardless of family. Size still mattered at matched correctness (rho = 0.80, p = 0.002), post-training changed reliance by at most 0.03 from 4B upward, and no model up to 14B solved the task internally, with OLMo 2 checkpoints never becoming capable. The task is a synthetic permutation exercise, so the scaling law is established there rather than on open-ended reasoning.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.