Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Meelis Kull, Peter A Flach

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

44 Citations (Scopus)
944 Downloads (Pure)

Abstract

There are several reasons to evaluate a multi-class classifier on other measures than just error rate. Perhaps most importantly, there can be uncertainty about the exact context of classifier deployment, requiring the classifier to perform well with respect to a variety of contexts. This is commonly achieved by creating a scoring classifier which outputs posterior class probability estimates. Proper scoring rules are loss evaluation measures of scoring classifiers which are minimised at the true posterior probabilities. The well-known decomposition of the proper scoring rules into calibration loss and refinement loss has facilitated the development of methods to reduce these losses, thus leading to better classifiers. We propose multiple novel decompositions including one with four terms: adjustment loss, post-adjustment calibration loss, grouping loss and irreducible loss. The separation of adjustment loss from calibration loss requires extra assumptions which we prove to be satisfied for the most frequently used proper scoring rules: Brier score and log-loss. We propose algorithms to perform adjustment as a simpler alternative to calibration.
Original languageEnglish
Title of host publicationMachine Learning and Knowledge Discovery in Databases
Subtitle of host publicationEuropean Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I
EditorsAnnalisa Appice, Pedro Pereira Rodrigues, Vítor Santos Costa, Carlos Soares, João Gama, Alípio Jorge
PublisherSpringer International Publishing AG
Pages68-85
Number of pages18
Volume1
ISBN (Electronic)9783319235288
ISBN (Print)9783319235271
DOIs
Publication statusPublished - 2015
EventEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - ECML PKDD PhD Consortium - Porto, Portugal, Portugal
Duration: 7 Sept 201511 Sept 2015

Publication series

NameLecture Notes in Computer Science
PublisherSpringer International Publishing
Volume9284
ISSN (Print)0302-9743

Conference

ConferenceEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases - ECML PKDD PhD Consortium
Country/TerritoryPortugal
CityPorto, Portugal
Period7/09/1511/09/15

Research Groups and Themes

  • Jean Golding

Fingerprint

Dive into the research topics of 'Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration'. Together they form a unique fingerprint.

Cite this