An agentic literature search matched 44 papers to clinical trials needing eligibility changes, and an LLM judge agreed with ten domain experts at a weighted kappa of 0.694
AspireAI community · arXiv
Overly restrictive eligibility criteria contribute to widespread enrollment failure in clinical trials, yet published recommendations for how specific criteria should be broadened remain disconnected from trial design at scale. RECITE (Revising Eligibility Criteria Incorporating Textual Evidence) is a framework for automated discovery, extraction and implementation of evidence-based eligibility criteria revisions. It uses protocol amendment documents from ClinicalTrials.gov to construct a dataset of 11,913 curated instances across 5,735 trials, each pairing original criteria with documentary evidence and target revisions. The authors benchmark fifteen LLM configurations under truncated-to-context-window and retrieval-augmented generation conditions on a 3,116-instance sample under an LLM-as-judge protocol validated against ten domain experts, with judge-versus-human quadratic-weighted Cohen's kappa of 0.694 plus or minus 0.096. The best open-weight model, Gemma 2 9B, achieves a kappa-aligned ordinal score of 3.0 plus or minus 0.9 on a 0 to 4 scale, normalized 0.742, and the best closed-weight model, GPT-4o, reaches 3.0 plus or minus 0.8, normalized 0.745. Using an agentic literature discovery system, the authors then identified 44 paper-trial matches across five publications providing trial-design and EC-modification evidence. The limitation is that the criterion-quality ceiling is set by the judge protocol itself, so the close open and closed weight scores mostly demonstrate that the evaluation is measuring something consistent rather than that either model can draft acceptable criteria autonomously.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.