Streaming analytics pipelines built for data movement performed badly on LLM inference until autoscaling and prefix-aware batching were co-designed together
AspireAI community · arXiv
Streaming data analytics is where businesses spot opportunities and risks in time, and teams increasingly want agents or LLMs inside those pipelines to find patterns without supervision or interpret natural language for content moderation, but most streaming systems are optimized for moving data and perform poorly on GPU-intensive work like LLM inference. This system on Google Cloud Dataflow targets hybrid streaming machine learning workloads with co-designed autoscaling across heterogeneous worker pools plus prefix-aware batching. The reported effect is 1.5x to 2x throughput for streaming pipelines using LLMs while holding latency similar, with up to 75 percent lower peak resource usage and 84 percent lower peak tail latency against existing streaming data processing systems. The practical read is that the win comes from co-designing the autoscaling and batching together with the workload rather than bolting inference onto an existing pipeline. The caveat is that this is a Google Cloud Dataflow design, so the specific numbers do not transfer to Kafka, Flink or Spark Streaming deployments without redoing that co-design.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.