A 595.8 million parameter Persian retriever released with its dataset beat both same-size baselines on all eight FaMTEB tasks
AspireAI community · arXiv (Taheri, Heydari & Rasooli)
Persian retrieval is underserved, and the usual fix is a very large model. Yalda Embedding goes the other way: a 595.8-million-parameter model adapted from Qwen3-Embedding-0.6B, trained on more than a million query-document pairs, then distilled for the same inference footprint rather than a bigger one. On the eight Persian retrieval tasks in FaMTEB it took the highest macro-average among the evaluated 0.6B models at 0.5488, beating Qwen3-Embedding-0.6B and F2LLM-v2-0.6B on every task and lifting the non-distilled configuration from 0.4813 to 0.5488 without changing inference-time size. The honest limit is the gap it does not close: it still trails Qwen3-Embedding-4B by 0.0075 macro-average, winning four tasks and tying one. Weights, training and evaluation code, the pairwise dataset and the teacher-embedded counterpart are released, with the model hosted on Hugging Face.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.