Projects per year
Abstract
For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce wellcalibrated estimates of the posterior probability. Isotonic calibration is a powerful nonparametric method that is however prone to overfitting on smaller datasets; hence a parametric method based on the logistic sigmoidal curve is commonly used. While logistic calibration is designed for normally distributed perclass scores, we demonstrate experimentally that many classifiers including Naive Bayes and Adaboost suffer from a particular distortion where these score distributions are heavily skewed. In such cases logistic calibration can easily yield probability estimates that are worse than the original scores. Moreover, the logistic curve family does not include the identity function, and hence logistic calibration can easily uncalibrate a perfectly calibrated classifier. In this paper we solve all these problems with a richer class of parametric calibration maps based on the beta distribution. We derive the method from first principles and show that fitting it is as easy as fitting a logistic curve. Extensive experiments show that beta calibration is superior to logistic calibration for a wide range of classifiers: Naive Bayes, Adaboost, random forest, logistic regression, support vector machine and multilayer perceptron. If the original classifier is already calibrated, then beta calibration learns a function close to the identity. On this we build a statistical test to recognise if the model deviates from being wellcalibrated.
Original language  English 

Pages (fromto)  50525080 
Number of pages  29 
Journal  Electronic Journal of Statistics 
Volume  11 
Issue number  2 
DOIs  
Publication status  Published  15 Dec 2017 
Structured keywords
 Digital Health
Keywords
 Beta distribution
 Binary classification
 Classifier calibration
 Logistic function
 Posterior probabilities
 Sigmoid
Fingerprint Dive into the research topics of 'Beyond Sigmoids: How to obtain wellcalibrated probabilities from binary classifiers with beta calibration'. Together they form a unique fingerprint.
Projects
 2 Finished

SPHERE (EPSRC IRC)
Craddock, I. J., Coyle, D. T., Flach, P. A., Kaleshi, D., Mirmehdi, M., Piechocki, R. J., Stark, B. H., Ascione, R., Ashburn, A. M., Burnett, M. E., Damen, D., GoobermanHill, R. J. S., Harwin, W. S., Hilton, G., Holderbaum, W., Holley, A. P., Manchester, V. A., Meller, B. J., Stack, E. & Gilchrist, I. D.
1/10/13 → 30/09/18
Project: Research, Parent

Activities
 1 Participation in workshop, seminar, course

Classifier Calibration
Peter A Flach (Organiser), Miquel Perello Nieto (Organiser), Hao Song (Organiser), Meelis Kull (Organiser) & Telmo De Menezes E Silva Filho (Organiser)
14 Sep 2020Activity: Participating in or organising an event types › Participation in workshop, seminar, course
Profiles

Professor Peter A Flach
 SPHERE
 Cabot Institute for the Environment
 Intelligent Systems Laboratory
 Department of Computer Science  Professor of Artificial Intelligence
Person: Academic , Member