Skip to content

Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy (Best paper award)

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part II
EditorsToon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
Publisher or commissioning bodySpringer Berlin Heidelberg
Number of pages16
ISBN (Electronic)9783662448519
ISBN (Print)9783662448502
DatePublished - 1 Sep 2014
EventEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France
Duration: 15 Sep 201419 Sep 2014

Publication series

NameLecture Notes in Artificial Intelligence
PublisherSpringer Berlin Heidelberg
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


ConferenceEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014


We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves. © 2014 Springer-Verlag.

    Structured keywords

  • Jean Golding


European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014

Duration15 Sep 201419 Sep 2014
SponsorsDeloitte (External organisation), EDF (External organisation), et al. (External organisation), Orange (External organisation), Winton (External organisation), Xerox Research Centre Europe (External organisation)

Event: Conference

Download statistics

No data available



  • Full-text PDF (accepted author manuscript)

    Rights statement: This is the author accepted manuscript (AAM). The final published version (version of record) is available online via Springer at Please refer to any applicable terms of use of the publisher.

    Accepted author manuscript, 486 KB, PDF document


View research connections

Related faculties, schools or groups