DAMI shows gated linear attention is bottlenecked by its initialization, not its architecture
AspireAI community · arXiv
Linear attention scales linearly with sequence length, but its finite-state memory can limit long-context retrieval: in gated linear recurrences, memory retention depends on state decay, which is shaped jointly by the gate parameters A_log and dt_bias and by input activations. In controlled Gated DeltaNet experiments the authors find both parameters remain highly correlated with their initial values after training - which points at initialization as the real bottleneck. Dual-Axis Memory Initialization (DAMI) jointly configures those parameters before training, assigning geometric timescales across heads for varied retrieval distances and a low-high-low input-sensitivity profile across layers, with a small activation sample calibrating the GDN-family implementation to match both behavioural targets without adding trainable parameters. Across GDN, KDA, GDN2, Mamba-2 and RWKV-7, the authors report DAMI improves average RULER-style retrieval accuracy by 5.29 to 16.71 percentage points over the corresponding random-initialization baselines, while performance on language modeling, commonsense reasoning and 2K real-world retrieval remains broadly comparable to those baselines. The limitation is that the gains are on RULER-style retrieval rather than long-document reasoning, and the paper is a preprint with no peer review noted.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.