Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)

Meelis Kull, Peter A. Flach

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

16 Citations (Scopus)
530 Downloads (Pure)

Abstract

We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves. © 2014 Springer-Verlag.

Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II
EditorsToon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
PublisherSpringer Berlin Heidelberg
Pages18-33
Number of pages16
ISBN (Electronic)9783662448519
ISBN (Print)9783662448502
DOIs
Publication statusPublished - 1 Sept 2014
EventEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France
Duration: 15 Sept 201419 Sept 2014

Publication series

NameLecture Notes in Artificial Intelligence
PublisherSpringer Berlin Heidelberg
Volume8725
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
Country/TerritoryFrance
CityNancy
Period15/09/1419/09/14

Research Groups and Themes

  • Jean Golding

Fingerprint

Dive into the research topics of 'Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)'. Together they form a unique fingerprint.

Cite this