Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)

Meelis Kull, Peter A. Flach

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

7 Citations (Scopus)
271 Downloads (Pure)

Abstract

We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves. © 2014 Springer-Verlag.

Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II
EditorsToon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
PublisherSpringer Berlin Heidelberg
Pages18-33
Number of pages16
ISBN (Electronic)9783662448519
ISBN (Print)9783662448502
DOIs
Publication statusPublished - 1 Sep 2014
EventEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France
Duration: 15 Sep 201419 Sep 2014

Publication series

NameLecture Notes in Artificial Intelligence
PublisherSpringer Berlin Heidelberg
Volume8725
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
CountryFrance
CityNancy
Period15/09/1419/09/14

Structured keywords

  • Jean Golding

Fingerprint Dive into the research topics of 'Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)'. Together they form a unique fingerprint.

  • Projects

    Prizes

    ECML-PKDD 2014 Best paper award

    Flach, Peter A (Recipient), 2014

    Prize: Prizes, Medals, Awards and Grants

    Cite this

    Kull, M., & Flach, P. A. (2014). Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award). In T. Calders, F. Esposito, E. Hullermeier, & R. Meo (Eds.), Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II (pp. 18-33). (Lecture Notes in Artificial Intelligence; Vol. 8725). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-44851-9_2