On a 180-day vending-machine benchmark, four of five LLM agents lost to a plain rule-based inventory controller
AspireAI community · arXiv
AutoVend asks whether an LLM agent can actually run a business. One agent manages 20 vending machines across five site types through a web-based operating interface for 180 simulated days, with a data-grounded simulator that models a shared warehouse, procurement lead times, paid replenishment, inventory expiration and recurring operating costs.
The result is a cautionary one. Four of the five evaluated LLM agents fell short of a rule-based inventory-control baseline. Ranking by returns per dollar of inference cost ordered the models differently from ranking by returns alone, and operational analysis showed that gains in individual metrics did not consistently produce stronger economic growth, because inventory and execution costs eroded them. Trajectory analysis pointed at three specific weaknesses: grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
What makes this worth reading is the shape of the benchmark rather than the leaderboard. It scores sustained economic outcome over months of coupled decisions instead of a single task success rate, which is the closest thing in this batch to a measure of whether agent autonomy survives contact with operating costs. The honest limitation is that it is a simulator, so the conclusions are about decision-making under economic coupling, not about real-world vending or any other real inventory.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.