Searching over data and model configurations jointly for LLM fine-tuning, without paying for a full training run at every step
AspireAI community · arXiv
Fine-tuning an LLM to maximise performance on a downstream task requires finding the optimal data and model configuration, which is a black-box optimisation problem that Bayesian optimisation handles by evaluating training configurations and using observed feedback to adaptively guide the search. Applying BO directly to the joint data-model space runs into two obstacles: the prohibitive cost of full LLM training at each iteration, and the large number of BO iterations needed for effective exploration given the dimensionality of the problem. JoBS is a BO-based approach that co-optimizes data and model configurations without requiring a full training run at every iteration. It allocates an initial portion of the optimisation budget to learning a scaling-law-inspired performance predictor that estimates fully trained LLM performance from only a small number of training steps, then runs BO with that predictor, which lets it explore significantly more configurations within the same budget. The paper analyses JoBS's average regret and derives the optimal budget allocation between predictor learning and BO iterations, and reports outperforming independent data and model optimisation methods and existing multi-fidelity BO baselines across a diverse set of downstream tasks, with code released. The limitation is that the headline gains are measured against the authors' own baselines on the search budgets they chose, and BO results are sensitive to the surrogate and acquisition setup.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.