Skip to content

Setting decision thresholds when operating conditions are uncertain

Research output: Contribution to journalArticle

Original languageEnglish
Pages (from-to)805-847
Number of pages43
JournalData Mining and Knowledge Discovery
Volume33
Issue number4
Early online date16 Feb 2019
DOIs
DateAccepted/In press - 3 Dec 2018
DateE-pub ahead of print - 16 Feb 2019
DatePublished (current) - 1 Jul 2019

Abstract

The quality of the decisions made by a machine learning model depends on the data and the operating conditions during deployment. Often, operating conditions such as class distribution and misclassification costs have changed during the time since the model was trained and evaluated. When deploying a binary classifier that outputs scores, once we know the new class distribution and the new cost ratio between false positives and false negatives, there are several methods in the literature to help us choose an appropriate threshold for the classifier’s scores. However, on many occasions, the information that we have about this operating condition is uncertain. Previous work has considered ranges or distributions of operating conditions during deployment, with expected costs being calculated for ranges or intervals, but still the decision for each point is made as if the operating condition were certain. The implications of this assumption have received limited attention: a threshold choice that is best suited without uncertainty may be suboptimal under uncertainty. In this paper we analyse the effect of operating condition uncertainty on the expected loss for different threshold choice methods, both theoretically and experimentally. We model uncertainty as a second conditional distribution over the actual operation condition and study it theoretically in such a way that minimum and maximum uncertainty are both seen as special cases of this general formulation. This is complemented by a thorough experimental analysis investigating how different learning algorithms behave for a range of datasets according to the threshold choice method and the uncertainty level.

    Research areas

  • Calibration, Classification, Operating condition, Threshold choice methods, Uncertainty

Download statistics

No data available

Documents

Documents

  • Full-text PDF (final published version)

    Rights statement: This is the final published version of the article (version of record). It first appeared online via Springer Link at https://doi.org/10.1007/s10618-019-00613-7 . Please refer to any applicable terms of use of the publisher.

    Final published version, 2.27 MB, PDF document

    Licence: CC BY

DOI

View research connections

Related faculties, schools or groups