Skip to content

Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II
EditorsToon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
Publisher or commissioning bodySpringer Berlin Heidelberg
Pages18-33
Number of pages16
ISBN (Electronic)9783662448519
ISBN (Print)9783662448502
DOIs
DatePublished - 1 Sep 2014
EventEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France
Duration: 15 Sep 201419 Sep 2014

Publication series

NameLecture Notes in Artificial Intelligence
PublisherSpringer Berlin Heidelberg
Volume8725
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
CountryFrance
CityNancy
Period15/09/1419/09/14

Abstract

We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves. © 2014 Springer-Verlag.

    Structured keywords

  • Jean Golding

Event

European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014

Duration15 Sep 201419 Sep 2014
CityNancy
CountryFrance
SponsorsDeloitte (External organisation), EDF (External organisation), et al. (External organisation), Orange (External organisation), Winton (External organisation), Xerox Research Centre Europe (External organisation)

Event: Conference

Download statistics

No data available

Documents

Documents

  • Full-text PDF (accepted author manuscript)

    Rights statement: This is the author accepted manuscript (AAM). The final published version (version of record) is available online via Springer at http://link.springer.com/chapter/10.1007%2F978-3-662-44851-9_2. Please refer to any applicable terms of use of the publisher.

    Accepted author manuscript, 486 KB, PDF document

DOI

View research connections

Related faculties, schools or groups