A four-step FLUX.1-dev model lost output diversity in its very first denoising step, and a 0.002 percent-parameter adapter recovered 48.3 percent of it
AspireAI community · arXiv
Distribution-matching distillation makes high-quality image generation possible in a handful of denoising steps, but the distilled student tends to map different initial noise samples onto nearly the same image for a given prompt. By comparing teacher and student along the sampling trajectory, the authors locate the loss: the largest reduction in response to different initial noises happens during the first, high-noise step, and swapping only that step back to the teacher's trajectory restores most of the diversity. ReSeed acts on that finding, adding a seed-response bias adapter to the first student step only and aligning its velocity with the teacher's first-interval secant velocity, leaving every student weight and later step untouched. On a distilled 4-step FLUX.1-dev model it needs 0.00196 percent additional trainable parameters relative to the base model, trains on 8,000 prompt-noise pairs with the student fully frozen and no target images, perceptual losses or adversarial losses, and improves DINO diversity by 48.3 percent while keeping competitive visual quality.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.