Rate-constrained ranking and the rate-weighted AUC

Louise A C Millard; Peter A. Flach; Julian P T Higgins

doi:10.1007/978-3-662-44851-9_25

Rate-constrained ranking and the rate-weighted AUC

Louise A C Millard, Peter A. Flach, Julian P T Higgins

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

4 Citations (Scopus)

Abstract

Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.

Original language	English
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Editors	Toon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
Publisher	Springer Verlag
Pages	386-403
Number of pages	18
Volume	8725 LNAI
Edition	PART 2
ISBN (Print)	9783662448502
DOIs	https://doi.org/10.1007/978-3-662-44851-9_25
Publication status	Published - 1 Jan 2014
Event	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France Duration: 15 Sep 2014 → 19 Sep 2014

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 2
Volume	8725 LNAI
ISSN (Print)	03029743
ISSN (Electronic)	16113349

Conference

Conference	European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
Country/Territory	France
City	Nancy
Period	15/09/14 → 19/09/14

Structured keywords

Jean Golding

Access to Document

10.1007/978-3-662-44851-9_25

Handle.net

Persistent link

Cite this

Millard, L. A. C., Flach, P. A., & Higgins, J. P. T. (2014). Rate-constrained ranking and the rate-weighted AUC. In T. Calders, F. Esposito, E. Hullermeier, & R. Meo (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (PART 2 ed., Vol. 8725 LNAI, pp. 386-403). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8725 LNAI, No. PART 2). Springer Verlag. https://doi.org/10.1007/978-3-662-44851-9_25

Millard, Louise A C ; Flach, Peter A. ; Higgins, Julian P T. / Rate-constrained ranking and the rate-weighted AUC. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). editor / Toon Calders ; Floriana Esposito ; Eyke Hullermeier ; Rosa Meo. Vol. 8725 LNAI PART 2. ed. Springer Verlag, 2014. pp. 386-403 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2).

@inproceedings{23f997b6be5e4088b5bf56c89cbe1133,

title = "Rate-constrained ranking and the rate-weighted AUC",

abstract = "Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. {\textcopyright} 2014 Springer-Verlag.",

author = "Millard, {Louise A C} and Flach, {Peter A.} and Higgins, {Julian P T}",

year = "2014",

month = jan,

day = "1",

doi = "10.1007/978-3-662-44851-9_25",

language = "English",

isbn = "9783662448502",

volume = "8725 LNAI",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

number = "PART 2",

pages = "386--403",

editor = "Toon Calders and Floriana Esposito and Eyke Hullermeier and Rosa Meo",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

address = "Germany",

edition = "PART 2",

note = "European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 ; Conference date: 15-09-2014 Through 19-09-2014",

}

Millard, LAC , Flach, PA & Higgins, JPT 2014, Rate-constrained ranking and the rate-weighted AUC. in T Calders, F Esposito, E Hullermeier & R Meo (eds), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 2 edn, vol. 8725 LNAI, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 2, vol. 8725 LNAI, Springer Verlag, pp. 386-403, European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014, Nancy, France, 15/09/14. https://doi.org/10.1007/978-3-662-44851-9_25

Rate-constrained ranking and the rate-weighted AUC. / Millard, Louise A C ; Flach, Peter A.; Higgins, Julian P T.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). ed. / Toon Calders; Floriana Esposito; Eyke Hullermeier; Rosa Meo. Vol. 8725 LNAI PART 2. ed. Springer Verlag, 2014. p. 386-403 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8725 LNAI, No. PART 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Rate-constrained ranking and the rate-weighted AUC

AU - Millard, Louise A C

AU - Flach, Peter A.

AU - Higgins, Julian P T

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.

AB - Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.

UR - http://www.scopus.com/inward/record.url?scp=84907042738&partnerID=8YFLogxK

U2 - 10.1007/978-3-662-44851-9_25

DO - 10.1007/978-3-662-44851-9_25

M3 - Conference Contribution (Conference Proceeding)

SN - 9783662448502

VL - 8725 LNAI

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 386

EP - 403

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

A2 - Calders, Toon

A2 - Esposito, Floriana

A2 - Hullermeier, Eyke

A2 - Meo, Rosa

PB - Springer Verlag

T2 - European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014

Y2 - 15 September 2014 through 19 September 2014

ER -

Millard LAC , Flach PA , Higgins JPT. Rate-constrained ranking and the rate-weighted AUC. In Calders T, Esposito F, Hullermeier E, Meo R, editors, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). PART 2 ed. Vol. 8725 LNAI. Springer Verlag. 2014. p. 386-403. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2). doi: 10.1007/978-3-662-44851-9_25

Rate-constrained ranking and the rate-weighted AUC

Abstract

Publication series

Conference

Structured keywords

Access to Document

Handle.net

Other files and links

Fingerprint

ConDuCT-II

IEU Theme 3

IEU Theme 2

Cite this

Rate-constrained ranking and the rate-weighted AUC

Abstract

Publication series

Conference

Structured keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Projects

ConDuCT-II

IEU Theme 3

IEU Theme 2

Cite this