Classifier calibration: a survey on how to assess and improve predicted class probabilities

Telmo de Menezes e Silva Filho, Hao Song, Miquel Perello Nieto*, Raul Santos-Rodriguez, Meelis Kull, Peter A Flach

*Corresponding author for this work

Research output: Contribution to journalArticle (Academic Journal)peer-review

54 Citations (Scopus)

Abstract

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.
Original languageEnglish
Pages (from-to)3211-3260
Number of pages50
JournalMachine Learning
Volume112
Issue number9
DOIs
Publication statusPublished - 16 May 2023

Bibliographical note

Funding Information:
The work of PF, RSR and MPN was supported by the SPHERE Next Steps Project funded by the UK Engineering and Physical Sciences Research Council (EPSRC) [grant EP/R005273/1]. The work of RSR was funded by the UKRI Turing AI Fellowship [grant EP/V024817/1]. The work of PF and HS was supported by The Alan Turing Institute under EPSRC [grant EP/N510129/1]. The work of PF was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. The work of MK was supported by Estonian Research Council [grant PRG1604] and by the Estonian Centre of Excellence in IT (EXCITE), funded by the European Regional Development Fund.

Publisher Copyright:
© 2023, The Author(s).

Keywords

  • Classification
  • Calibration
  • Confidence
  • Uncertainty
  • Multiclass
  • Evaluation

Fingerprint

Dive into the research topics of 'Classifier calibration: a survey on how to assess and improve predicted class probabilities'. Together they form a unique fingerprint.

Cite this