Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Miquel Perello-Nieto; Telmo M Silva Filho; Meelis Kull; Peter Flach

doi:10.1109/ICDM.2016.0150

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Miquel Perello-Nieto, Telmo M Silva Filho, Meelis Kull, Peter Flach

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

12 Citations (Scopus)

716 Downloads (Pure)

Abstract

We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option; (ii) identify outliers; and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions. One of these assumes an affine relationship with two parameters, and we show how this bivariate parameter space naturally interpolates between the above capabilities.We demonstrate the versatility of the approach by comparing it experimentally with published special-purpose solutions for outlier detection and confident classification on 41 benchmark datasets. Results show that Background Check can match and in many cases surpass the performances of specialised approaches.

Original language	English
Title of host publication	2016 IEEE 16th International Conference on Data Mining (ICDM 2016)
Subtitle of host publication	Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	1143-1148
Number of pages	6
ISBN (Electronic)	9781509054725
ISBN (Print)	9781509054749
DOIs	https://doi.org/10.1109/ICDM.2016.0150
Publication status	Published - Mar 2017
Event	16th IEEE International Conference on Data Mining, ICDM 2016 - Barcelona, Catalonia, Spain Duration: 12 Dec 2016 → 15 Dec 2016

Publication series

Name	Proceedings of the IEEE International Conference on Data Mining (ICDM)
Publisher	IEEE
ISSN (Print)	2375-8486

Conference

Conference	16th IEEE International Conference on Data Mining, ICDM 2016
Country/Territory	Spain
City	Barcelona, Catalonia
Period	12/12/16 → 15/12/16

Research Groups and Themes

Jean Golding
SPHERE

Keywords

Training data
Estimation
Standards
Reliability
Data models
Probability
Conferences
Multiclass classification
Outlier detection
Confident classification

Access to Document

10.1109/ICDM.2016.0150

Full-text PDF (accepted author manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via IEEE at http://ieeexplore.ieee.org/document/7837963/ . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 201 KB

Handle.net

Persistent link

Cite this

Perello-Nieto, M., Filho, T. M. S., Kull, M., & Flach, P. (2017). Background Check: A General Technique to Build More Reliable and Versatile Classifiers. In 2016 IEEE 16th International Conference on Data Mining (ICDM 2016): Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain (pp. 1143-1148). Article 7837963 (Proceedings of the IEEE International Conference on Data Mining (ICDM)). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICDM.2016.0150

Perello-Nieto, Miquel ; Filho, Telmo M Silva ; Kull, Meelis et al. / Background Check : A General Technique to Build More Reliable and Versatile Classifiers. 2016 IEEE 16th International Conference on Data Mining (ICDM 2016): Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain. Institute of Electrical and Electronics Engineers (IEEE), 2017. pp. 1143-1148 (Proceedings of the IEEE International Conference on Data Mining (ICDM)).

@inproceedings{fa9afc9d6a7b48639f4ac889eea825cc,

title = "Background Check: A General Technique to Build More Reliable and Versatile Classifiers",

abstract = "We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option; (ii) identify outliers; and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions. One of these assumes an affine relationship with two parameters, and we show how this bivariate parameter space naturally interpolates between the above capabilities.We demonstrate the versatility of the approach by comparing it experimentally with published special-purpose solutions for outlier detection and confident classification on 41 benchmark datasets. Results show that Background Check can match and in many cases surpass the performances of specialised approaches.",

keywords = "Training data, Estimation, Standards, Reliability, Data models, Probability, Conferences, Multiclass classification, Outlier detection, Confident classification",

author = "Miquel Perello-Nieto and Filho, {Telmo M Silva} and Meelis Kull and Peter Flach",

year = "2017",

month = mar,

doi = "10.1109/ICDM.2016.0150",

language = "English",

isbn = "9781509054749",

series = "Proceedings of the IEEE International Conference on Data Mining (ICDM)",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "1143--1148",

booktitle = "2016 IEEE 16th International Conference on Data Mining (ICDM 2016)",

address = "United States",

note = "16th IEEE International Conference on Data Mining, ICDM 2016 ; Conference date: 12-12-2016 Through 15-12-2016",

}

Perello-Nieto, M, Filho, TMS, Kull, M & Flach, P 2017, Background Check: A General Technique to Build More Reliable and Versatile Classifiers. in 2016 IEEE 16th International Conference on Data Mining (ICDM 2016): Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain., 7837963, Proceedings of the IEEE International Conference on Data Mining (ICDM), Institute of Electrical and Electronics Engineers (IEEE), pp. 1143-1148, 16th IEEE International Conference on Data Mining, ICDM 2016, Barcelona, Catalonia, Spain, 12/12/16. https://doi.org/10.1109/ICDM.2016.0150

Background Check: A General Technique to Build More Reliable and Versatile Classifiers. / Perello-Nieto, Miquel; Filho, Telmo M Silva; Kull, Meelis et al.
2016 IEEE 16th International Conference on Data Mining (ICDM 2016): Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain. Institute of Electrical and Electronics Engineers (IEEE), 2017. p. 1143-1148 7837963 (Proceedings of the IEEE International Conference on Data Mining (ICDM)).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Background Check

T2 - 16th IEEE International Conference on Data Mining, ICDM 2016

AU - Perello-Nieto, Miquel

AU - Filho, Telmo M Silva

AU - Kull, Meelis

AU - Flach, Peter

PY - 2017/3

Y1 - 2017/3

N2 - We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option; (ii) identify outliers; and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions. One of these assumes an affine relationship with two parameters, and we show how this bivariate parameter space naturally interpolates between the above capabilities.We demonstrate the versatility of the approach by comparing it experimentally with published special-purpose solutions for outlier detection and confident classification on 41 benchmark datasets. Results show that Background Check can match and in many cases surpass the performances of specialised approaches.

AB - We introduce a powerful technique to make classifiers more reliable and versatile. Background Check equips classifiers with the ability to assess the difference of unlabelled test data from the training data. In particular, Background Check gives classifiers the capability to (i) perform cautious classification with a reject option; (ii) identify outliers; and (iii) better assess the confidence in their predictions. We derive the method from first principles and consider four particular relationships between background and foreground distributions. One of these assumes an affine relationship with two parameters, and we show how this bivariate parameter space naturally interpolates between the above capabilities.We demonstrate the versatility of the approach by comparing it experimentally with published special-purpose solutions for outlier detection and confident classification on 41 benchmark datasets. Results show that Background Check can match and in many cases surpass the performances of specialised approaches.

KW - Training data

KW - Estimation

KW - Standards

KW - Reliability

KW - Data models

KW - Probability

KW - Conferences

KW - Multiclass classification

KW - Outlier detection

KW - Confident classification

UR - http://www.scopus.com/inward/record.url?scp=85014572175&partnerID=8YFLogxK

U2 - 10.1109/ICDM.2016.0150

DO - 10.1109/ICDM.2016.0150

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85014572175

SN - 9781509054749

T3 - Proceedings of the IEEE International Conference on Data Mining (ICDM)

SP - 1143

EP - 1148

BT - 2016 IEEE 16th International Conference on Data Mining (ICDM 2016)

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 12 December 2016 through 15 December 2016

ER -

Perello-Nieto M, Filho TMS, Kull M, Flach P. Background Check: A General Technique to Build More Reliable and Versatile Classifiers. In 2016 IEEE 16th International Conference on Data Mining (ICDM 2016): Proceedings of a meeting held 12-15 December 2016, Barcelona, Spain. Institute of Electrical and Electronics Engineers (IEEE). 2017. p. 1143-1148. 7837963. (Proceedings of the IEEE International Conference on Data Mining (ICDM)). Epub 2017 Feb 2. doi: 10.1109/ICDM.2016.0150

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Abstract

Publication series

Conference

Research Groups and Themes

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

SPHERE (EPSRC IRC)

REFrAMe

Uncertainty aware classification: augmenting classifiers to handle uncertainty

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Abstract

Publication series

Conference

Research Groups and Themes

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Projects

SPHERE (EPSRC IRC)

REFrAMe

Student theses

Uncertainty aware classification: augmenting classifiers to handle uncertainty

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this