Setting decision thresholds when operating conditions are uncertain

Cèsar Ferri, José Hernández-Orallo, Peter Flach*

*Corresponding author for this work

Research output: Contribution to journalArticle (Academic Journal)peer-review

150 Downloads (Pure)

Abstract

The quality of the decisions made by a machine learning model depends on the data and the operating conditions during deployment. Often, operating conditions such as class distribution and misclassification costs have changed during the time since the model was trained and evaluated. When deploying a binary classifier that outputs scores, once we know the new class distribution and the new cost ratio between false positives and false negatives, there are several methods in the literature to help us choose an appropriate threshold for the classifier’s scores. However, on many occasions, the information that we have about this operating condition is uncertain. Previous work has considered ranges or distributions of operating conditions during deployment, with expected costs being calculated for ranges or intervals, but still the decision for each point is made as if the operating condition were certain. The implications of this assumption have received limited attention: a threshold choice that is best suited without uncertainty may be suboptimal under uncertainty. In this paper we analyse the effect of operating condition uncertainty on the expected loss for different threshold choice methods, both theoretically and experimentally. We model uncertainty as a second conditional distribution over the actual operation condition and study it theoretically in such a way that minimum and maximum uncertainty are both seen as special cases of this general formulation. This is complemented by a thorough experimental analysis investigating how different learning algorithms behave for a range of datasets according to the threshold choice method and the uncertainty level.

Original languageEnglish
Pages (from-to)805-847
Number of pages43
JournalData Mining and Knowledge Discovery
Volume33
Issue number4
Early online date16 Feb 2019
DOIs
Publication statusPublished - 1 Jul 2019

Keywords

  • Calibration
  • Classification
  • Operating condition
  • Threshold choice methods
  • Uncertainty

Fingerprint Dive into the research topics of 'Setting decision thresholds when operating conditions are uncertain'. Together they form a unique fingerprint.

Cite this