The critic a reinforcement-learned model already has turned out to be a usable confidence estimator, with no retraining

Confidence estimation is usually treated as a separate training stage with its own data and compute. This paper asks what that stage actually solves and finds an answer hiding in plain sight. Comparing self- and external estimators of comparable capacity trained on the same question-response-correctness triples, the authors show an external model trained only on a policy's behavioral traces can estimate that policy's confidence as accurately as the policy itself - without ever seeing the policy's internal representations. The theory makes the connection explicit: under a correctness-based outcome, confidence estimation and critic training are equivalent, both predicting the likelihood that a response will be correct from its trajectory. The practical consequence is that for models trained with value-based reinforcement learning, the critic already sitting in the training stack can be reused as an off-the-shelf confidence estimator, with no additional training or data. The policy is untouched, so task performance is preserved, and the critic supplies confidence at intermediate reasoning steps too. Across math, code, tool calling and general reasoning spanning a dozen domains, the reused critic was the best confidence estimator on every in-distribution split and the best-calibrated on 14 of 16. Caveat: it depends on the model having been trained with a value-based RL critic to begin with, and the evaluation covers in-distribution splits.