Property Finder found 96.8% of its vision-model evaluation labels were copied from the incumbent model's own predictions, then rebuilt ground truth and cut inference cost 86% by moving to Llama 4 Maverick on Bedrock
AspireAI community · Amazon Web Services
An October 27, 2026 AWS post documents a methodology problem that applies to any team comparing models: Property Finder had been generating evaluation labels by running its production model and treating the outputs as ground truth. AWS reports that 96.8% of labeled samples came directly from the existing vision language model's predictions, so any alternative was structurally disadvantaged before a single image was processed — and the hidden errors were real, including hallucinated brand names in watermark text extraction and single labels forced onto images showing multiple room types. The fix was fresh expert annotation with no reference to prior model output: 356 hand-annotated UAE images, a 256-image development split and a held-out evaluation set, plus a switch from single-label to multi-label classification. On top of that, 27 prompt versions were iterated with error analysis, MIPROv2 and GEPA-style evolutionary optimization. The final Llama 4 Maverick 17B Instruct prompt scored 0.80 Micro F1 on the 356-image evaluation set against proprietary model A's 0.56, at an estimated $2.35 per 1,000 images and an 86% cost reduction, with Property Finder reporting 40–60% lower inference latency. The honest limitation: AWS states these scores are optimistic because both models' prompts were tuned on the same 256 images that sit inside the evaluation set, and the cost figures are estimates dated December 2025.
Read the story and explore more from the AspireAI community.
Two sparse-autoencoder analyses have resisted scaling: matching semantically similar features across layers, and compressing large feature circuits into interpretable supernodes. This paper argues both are the same underlying problem, estimating semantic distances between SAE features that sit on different activation manifolds. Its representation treats each feature not as a single decoder vector but as an activation-weighted distribution over the hidden states expressing it, projects those distributions into a shared reference space, and compares them with Wasserstein distance. The representation is proved invariant to activation rescaling and stable under perturbation, and recovers true matches under finite-sample margin conditions. Empirically it outperformed decoder-vector and LLM-based baselines and captured subtle functional distinctions between related features, and it compressed large feature circuits into interpretable supernodes automatically. The limitation is that this is interpretability tooling: the paper demonstrates better feature matching and circuit compression, not that downstream interpretability claims become more accurate.