LinkedIn replaced list-style LLM labeling with prefill-only scoring, raising throughput 1.9x in serving while changing at most 0.01 percent of decisions
AspireAI community · arXiv
A common way to use an LLM as a classifier is to pack a hundred candidate labels into one prompt and ask for a YES/NO list. The decisions in that list are independent - each one depends only on the entity and a single candidate - but the model has to decode the whole list, keep the list well-formed, and the answers are coupled to each other by construction. LinkedIn's paper rewrites the problem as scoring instead of generation: fine-tune on single-candidate prompts and read the decision off a single answer token in a prefill-only pass, with attention masked between candidates so a batch of them can be scored in one sequence.
The deployed result is a throughput story with a quality floor under it. Semantic scoring changed at most 0.01 percent of decisions when the candidate order was shuffled, matched the agreement with reference labels that the packed-generation approach only reached with about 4.7 times as many training pairs, and raised throughput 1.24x offline and 1.9x in serving against the previously deployed stack under a one-second p99 latency target. The MIS implementation was written as a reusable Ray Data API and the serving support was contributed upstream to SGLang.
The caveat is the shape of the task. Profile qualification is a binary call per candidate, which is precisely where scoring and generation are interchangeable; a labelling problem that needs a written justification is a different question. The throughput ratios are also relative to LinkedIn's own stack rather than an absolute rate.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.