Fixing LLM Overconfidence: A Deep Dive Into Temperature, Platt, and Isotonic Scaling

This article explores how temperature scaling, Platt scaling, and isotonic regression can recalibrate confidence scores in large language models to better reflect actual accuracy. It examines the challenges LLMs pose to classical calibration methods, including exponential output spaces, API limitations, and RLHF-induced overconfidence. The piece details how each technique works, its specific limitations in generative contexts, and modern variants like Adaptive Temperature Scaling and Multivariate Platt Scaling that address these gaps.