Beyond Sigmoids : How to obtain well-calibrated probabilities from binary classifiers with beta calibration

Meelis Kull, Telmo M. Silva Filho, Peter Flach

Research output: Contribution to journalArticle (Academic Journal)peer-review

76 Citations (Scopus)
3228 Downloads (Pure)

Abstract

For optimal decision making under variable class distributions and misclassification costs a classifier needs to produce well-calibrated estimates of the posterior probability. Isotonic calibration is a powerful non-parametric method that is however prone to overfitting on smaller datasets; hence a parametric method based on the logistic sigmoidal curve is commonly used. While logistic calibration is designed for normally distributed per-class scores, we demonstrate experimentally that many classifiers including Naive Bayes and Adaboost suffer from a particular distortion where these score distributions are heavily skewed. In such cases logistic calibration can easily yield probability estimates that are worse than the original scores. Moreover, the logistic curve family does not include the identity function, and hence logistic calibration can easily uncalibrate a perfectly calibrated classifier. In this paper we solve all these problems with a richer class of parametric calibration maps based on the beta distribution. We derive the method from first principles and show that fitting it is as easy as fitting a logistic curve. Extensive experiments show that beta calibration is superior to logistic calibration for a wide range of classifiers: Naive Bayes, Adaboost, random forest, logistic regression, support vector machine and multi-layer perceptron. If the original classifier is already calibrated, then beta calibration learns a function close to the identity. On this we build a statistical test to recognise if the model deviates from being well-calibrated.

Original languageEnglish
Pages (from-to)5052-5080
Number of pages29
JournalElectronic Journal of Statistics
Volume11
Issue number2
DOIs
Publication statusPublished - 15 Dec 2017

Research Groups and Themes

  • Digital Health
  • SPHERE

Keywords

  • Beta distribution
  • Binary classification
  • Classifier calibration
  • Logistic function
  • Posterior probabilities
  • Sigmoid

Fingerprint

Dive into the research topics of 'Beyond Sigmoids : How to obtain well-calibrated probabilities from binary classifiers with beta calibration'. Together they form a unique fingerprint.
  • SPHERE (EPSRC IRC)

    Craddock, I. J. (Principal Investigator), Coyle, D. T. (Principal Investigator), Flach, P. A. (Principal Investigator), Kaleshi, D. (Principal Investigator), Mirmehdi, M. (Principal Investigator), Piechocki, R. J. (Principal Investigator), Stark, B. H. (Principal Investigator), Ascione, R. (Co-Principal Investigator), Ashburn, A. M. (Collaborator), Burnett, M. E. (Collaborator), Damen, D. (Co-Principal Investigator), Gooberman-Hill, R. (Principal Investigator), Harwin, W. S. (Collaborator), Hilton, G. (Co-Principal Investigator), Holderbaum, W. (Collaborator), Holley, A. P. (Manager), Manchester, V. A. (Administrator), Meller, B. J. (Other ), Stack, E. (Collaborator) & Gilchrist, I. D. (Principal Investigator)

    1/10/1330/09/18

    Project: Research, Parent

  • REFrAMe

    Flach, P. A. (Principal Investigator)

    Engineering and Physical Sciences Research Council

    1/02/131/08/16

    Project: Research

Cite this