LLM-PowerTrace released per-GPU power traces sampled every 100 milliseconds, because joules-per-token hides how AI compute actually loads a grid
AspireAI community · arXiv (LLM-PowerTrace)
Energy benchmarks for large language models usually report aggregates such as joules per token and rarely release the underlying time series, so peaks, ramps and synchronized fluctuations across accelerators stay uncharacterized. LLM-PowerTrace is an open dataset of per-GPU power traces for LLM fine-tuning and inference sampled every 100 milliseconds on varying NVIDIA platforms, with full workload and system metadata. It sweeps batch size, sequence length, precision, parallelism and checkpointing for fine-tuning, and concurrency, GPU frequency, KV-cache and weight quantization, prompt and output lengths and request-arrival patterns for inference. On top of the traces the authors define power-grid-facing metrics and an AI-compute power-and-energy scaling law for extrapolating past the measured settings, which is what makes the dataset useful for forecasting and managing AI compute's electrical behaviour.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.