Classifier calibration: a survey on how to assess and improve predicted class probabilities

Telmo de Menezes e Silva Filho; Hao Song; Miquel Perello Nieto; Raul Santos-Rodriguez; Meelis Kull; Peter A Flach

doi:10.1007/s10994-023-06336-7

Classifier calibration: a survey on how to assess and improve predicted class probabilities

Telmo de Menezes e Silva Filho, Hao Song, Miquel Perello Nieto^*, Raul Santos-Rodriguez, Meelis Kull, Peter A Flach

^*Corresponding author for this work

Research output: Contribution to journal › Article (Academic Journal) › peer-review

54 Citations (Scopus)

Abstract

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

Original language	English
Pages (from-to)	3211-3260
Number of pages	50
Journal	Machine Learning
Volume	112
Issue number	9
DOIs	https://doi.org/10.1007/s10994-023-06336-7
Publication status	Published - 16 May 2023

Bibliographical note

Funding Information:
The work of PF, RSR and MPN was supported by the SPHERE Next Steps Project funded by the UK Engineering and Physical Sciences Research Council (EPSRC) [grant EP/R005273/1]. The work of RSR was funded by the UKRI Turing AI Fellowship [grant EP/V024817/1]. The work of PF and HS was supported by The Alan Turing Institute under EPSRC [grant EP/N510129/1]. The work of PF was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. The work of MK was supported by Estonian Research Council [grant PRG1604] and by the Estonian Centre of Excellence in IT (EXCITE), funded by the European Regional Development Fund.

Publisher Copyright:
© 2023, The Author(s).

Keywords

Classification
Calibration
Confidence
Uncertainty
Multiclass
Evaluation

Access to Document

10.1007/s10994-023-06336-7Licence: CC BY

Handle.net

Persistent link

54 Citations
4 Conference Contribution (Conference Proceeding)
1 Article (Academic Journal)

Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
Kull, M., Perello Nieto, M., Kängsepp, M., Silva Filho, T., Song, H. & Flach, P., 14 Dec 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019). Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E. & Garnett, R. (eds.). Neural Information Processing Systems Foundation, 12 p.
Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Open Access
File
Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers
Kull, M., De Menezes E Silva Filho, T. & Flach, P., 1 Apr 2017, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS 2017). Journal of Machine Learning Research, 9 p. (JMLR Workshop and Conference Proceedings; vol. 54).
Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Open Access
File
Beyond Sigmoids : How to obtain well-calibrated probabilities from binary classifiers with beta calibration
Kull, M., Silva Filho, T. M. & Flach, P., 15 Dec 2017, In: Electronic Journal of Statistics. 11, 2, p. 5052-5080 29 p.
Research output: Contribution to journal › Article (Academic Journal) › peer-review

Open Access
File
67 Citations (Scopus)

3180 Downloads (Pure)

1 Active
2 Finished

8032 EPSRC Turing AI Fellowship: Interactive Annotations in AI EP/V024817/1
Santos-Rodriguez, R. (Principal Investigator)
1/01/21 → 31/12/25
Project: Research
TAILOR: 8030 H2020 TAILOR 952215
Flach, P. A. (Principal Investigator)
1/09/20 → 31/08/23
Project: Research, Parent
SPHERE2
Craddock, I. J. (Principal Investigator), Mirmehdi, M. (Co-Investigator), Piechocki, R. J. (Co-Investigator), Flach, P. A. (Co-Investigator), Oikonomou, G. (Co-Investigator), Burghardt, T. (Co-Investigator), Damen, D. (Co-Investigator), Santos-Rodriguez, R. (Co-Investigator), O'Kane, A. A. (Co-Investigator), McConville, R. (Co-Investigator), Masullo, A. (Co-Investigator) & Gooberman-Hill, R. (Co-Investigator)
1/10/18 → 31/01/23
Project: Research, Parent

Uncertainty aware classification: augmenting classifiers to handle uncertainty
Perello Nieto, M. (Author), Flach, P. (Supervisor) & Santos-Rodriguez, R. (Supervisor), 9 May 2023
Student thesis: Doctoral Thesis › Doctor of Philosophy (PhD)

File

1 Participation in workshop, seminar, course

Classifier Calibration
Flach, P. A. (Organiser), Perello Nieto, M. (Organiser), Song, H. (Organiser), Kull, M. (Organiser) & De Menezes E Silva Filho, T. (Organiser)
14 Sept 2020
Activity: Participating in or organising an event types › Participation in workshop, seminar, course

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

@article{bd4218e285df493b919e910e5958644d,

title = "Classifier calibration: a survey on how to assess and improve predicted class probabilities",

abstract = "This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.",

keywords = "Classification, Calibration, Confidence, Uncertainty, Multiclass, Evaluation",

author = "{de Menezes e Silva Filho}, Telmo and Hao Song and {Perello Nieto}, Miquel and Raul Santos-Rodriguez and Meelis Kull and Flach, {Peter A}",

note = "Funding Information: The work of PF, RSR and MPN was supported by the SPHERE Next Steps Project funded by the UK Engineering and Physical Sciences Research Council (EPSRC) [grant EP/R005273/1]. The work of RSR was funded by the UKRI Turing AI Fellowship [grant EP/V024817/1]. The work of PF and HS was supported by The Alan Turing Institute under EPSRC [grant EP/N510129/1]. The work of PF was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. The work of MK was supported by Estonian Research Council [grant PRG1604] and by the Estonian Centre of Excellence in IT (EXCITE), funded by the European Regional Development Fund. Publisher Copyright: {\textcopyright} 2023, The Author(s).",

year = "2023",

month = may,

day = "16",

doi = "10.1007/s10994-023-06336-7",

language = "English",

volume = "112",

pages = "3211--3260",

journal = "Machine Learning",

issn = "0885-6125",

publisher = "Springer Verlag",

number = "9",

}

TY - JOUR

T1 - Classifier calibration

T2 - a survey on how to assess and improve predicted class probabilities

AU - de Menezes e Silva Filho, Telmo

AU - Song, Hao

AU - Perello Nieto, Miquel

AU - Santos-Rodriguez, Raul

AU - Kull, Meelis

AU - Flach, Peter A

N1 - Funding Information: The work of PF, RSR and MPN was supported by the SPHERE Next Steps Project funded by the UK Engineering and Physical Sciences Research Council (EPSRC) [grant EP/R005273/1]. The work of RSR was funded by the UKRI Turing AI Fellowship [grant EP/V024817/1]. The work of PF and HS was supported by The Alan Turing Institute under EPSRC [grant EP/N510129/1]. The work of PF was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. The work of MK was supported by Estonian Research Council [grant PRG1604] and by the Estonian Centre of Excellence in IT (EXCITE), funded by the European Regional Development Fund. Publisher Copyright: © 2023, The Author(s).

PY - 2023/5/16

Y1 - 2023/5/16

N2 - This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

AB - This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

KW - Classification

KW - Calibration

KW - Confidence

KW - Uncertainty

KW - Multiclass

KW - Evaluation

U2 - 10.1007/s10994-023-06336-7

DO - 10.1007/s10994-023-06336-7

M3 - Article (Academic Journal)

SN - 0885-6125

VL - 112

SP - 3211

EP - 3260

JO - Machine Learning

JF - Machine Learning

IS - 9

ER -

Classifier calibration: a survey on how to assess and improve predicted class probabilities

Abstract

Bibliographical note

Keywords

Access to Document

Handle.net

Fingerprint

Research output

Projects

Student theses

Activities

Equipment

Cite this