Cross-modal steering vectors transferred zero-shot in both directions across unified multimodal models, recovering 84% of the image-steering effect and 74% of the text-steering effect from a vector found in the other modality
AspireAI community · arXiv (Apple / academic collaboration)
Cross-modal Steering Vectors Exist Across Both Image and Text Generation, an arXiv paper submitted October 26, 2026, operationalizes the question of whether AI models implement concepts once for all modalities or separately per modality, and does so causally rather than through correlational representational-similarity analysis. The method finds a vector that steers generation on tokens of one modality and tests whether adding it to tokens of the other modality transfers zero-shot. Across unified multimodal models with shared language and image parameters, and across concepts including emotion and object size, steering vectors transfer in both directions, with the authors reporting that at middle layers a transferred vector recovers on average 84% of the intra-modality steering effect on images and 74% on text, and that for every concept except spatial there is a layer range where text-derived vectors recover more than 80% of the intra-modality effect across all three models. Concepts highly specific to one modality still transfer, with formal-versus-colloquial language steering producing formal clothing and design in the other modality, and the transfer has a geometric signature: despite a clear modality gap between text and image representations, strongly transferable vectors point in similar directions for the same concept. The limitation is that steering is inherently lossy - image coherence stays at about 90% of its unsteered level at the strongest tested setting, and the authors note a few failed edits were regenerated with a stronger model. Code is available at github.com/saintsauceee/cross-modal-steering.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.