Rate-oriented point-wise confidence bounds for ROC curves

Louise A C Millard; Meelis Kull; Peter A. Flach

doi:10.1007/978-3-662-44851-9_26

Rate-oriented point-wise confidence bounds for ROC curves

Louise A C Millard, Meelis Kull, Peter A. Flach

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

3 Citations (Scopus)

Abstract

Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new 'rate-oriented' approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value. © 2014 Springer-Verlag.

Original language	English
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Editors	Toon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
Publisher	Springer Berlin Heidelberg
Pages	404-421
Number of pages	18
Volume	8725 LNAI
Edition	PART 2
ISBN (Print)	9783662448502
DOIs	https://doi.org/10.1007/978-3-662-44851-9_26
Publication status	Published - 1 Jan 2014
Event	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France Duration: 15 Sept 2014 → 19 Sept 2014

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 2
Volume	8725 LNAI
ISSN (Print)	03029743
ISSN (Electronic)	16113349

Conference

Conference	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
Country/Territory	France
City	Nancy
Period	15/09/14 → 19/09/14

Structured keywords

Jean Golding

Keywords

Confidence bounds
rate-averaging
ROC curves
rate-constrained

Access to Document

10.1007/978-3-662-44851-9_26

http://dx.doi.org/10.1007/978-3-662-44851-9_26

Handle.net

Persistent link

IEU Theme 2
Flach, P. A., Gaunt, T. R. & Gaunt, T. R.
1/06/13 → 31/03/18
Project: Research
REFrAMe
Flach, P. A.
Engineering and Physical Sciences Research Council
1/02/13 → 1/08/16
Project: Research

Cite this

Millard, L. A. C., Kull, M., & Flach, P. A. (2014). Rate-oriented point-wise confidence bounds for ROC curves. In T. Calders, F. Esposito, E. Hullermeier, & R. Meo (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (PART 2 ed., Vol. 8725 LNAI, pp. 404-421). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8725 LNAI, No. PART 2). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-44851-9_26

Millard, Louise A C ; Kull, Meelis ; Flach, Peter A. / Rate-oriented point-wise confidence bounds for ROC curves. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). editor / Toon Calders ; Floriana Esposito ; Eyke Hullermeier ; Rosa Meo. Vol. 8725 LNAI PART 2. ed. Springer Berlin Heidelberg, 2014. pp. 404-421 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2).

@inproceedings{7f005e7d732147d19d1eecc5aa503508,

title = "Rate-oriented point-wise confidence bounds for ROC curves",

abstract = "Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new 'rate-oriented' approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value. {\textcopyright} 2014 Springer-Verlag.",

keywords = "Confidence bounds, rate-averaging, ROC curves, rate-constrained",

author = "Millard, {Louise A C} and Meelis Kull and Flach, {Peter A.}",

year = "2014",

month = jan,

day = "1",

doi = "10.1007/978-3-662-44851-9_26",

language = "English",

isbn = "9783662448502",

volume = "8725 LNAI",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Berlin Heidelberg",

number = "PART 2",

pages = "404--421",

editor = "Toon Calders and Floriana Esposito and Eyke Hullermeier and Rosa Meo",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

address = "Germany",

edition = "PART 2",

note = "European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 ; Conference date: 15-09-2014 Through 19-09-2014",

}

Millard, LAC, Kull, M & Flach, PA 2014, Rate-oriented point-wise confidence bounds for ROC curves. in T Calders, F Esposito, E Hullermeier & R Meo (eds), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 2 edn, vol. 8725 LNAI, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 2, vol. 8725 LNAI, Springer Berlin Heidelberg, pp. 404-421, European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014, Nancy, France, 15/09/14. https://doi.org/10.1007/978-3-662-44851-9_26

Rate-oriented point-wise confidence bounds for ROC curves. / Millard, Louise A C; Kull, Meelis; Flach, Peter A.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). ed. / Toon Calders; Floriana Esposito; Eyke Hullermeier; Rosa Meo. Vol. 8725 LNAI PART 2. ed. Springer Berlin Heidelberg, 2014. p. 404-421 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8725 LNAI, No. PART 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Rate-oriented point-wise confidence bounds for ROC curves

AU - Millard, Louise A C

AU - Kull, Meelis

AU - Flach, Peter A.

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new 'rate-oriented' approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value. © 2014 Springer-Verlag.

AB - Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new 'rate-oriented' approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value. © 2014 Springer-Verlag.

KW - Confidence bounds

KW - rate-averaging

KW - ROC curves

KW - rate-constrained

U2 - 10.1007/978-3-662-44851-9_26

DO - 10.1007/978-3-662-44851-9_26

M3 - Conference Contribution (Conference Proceeding)

SN - 9783662448502

VL - 8725 LNAI

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 404

EP - 421

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

A2 - Calders, Toon

A2 - Esposito, Floriana

A2 - Hullermeier, Eyke

A2 - Meo, Rosa

PB - Springer Berlin Heidelberg

T2 - European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014

Y2 - 15 September 2014 through 19 September 2014

ER -

Millard LAC, Kull M, Flach PA. Rate-oriented point-wise confidence bounds for ROC curves. In Calders T, Esposito F, Hullermeier E, Meo R, editors, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 2 ed. Vol. 8725 LNAI. Springer Berlin Heidelberg. 2014. p. 404-421. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2). doi: 10.1007/978-3-662-44851-9_26

Rate-oriented point-wise confidence bounds for ROC curves

Abstract

Publication series

Conference

Structured keywords

Keywords

Access to Document

Handle.net

Fingerprint

Projects

IEU Theme 2

REFrAMe

Cite this