Projects per year
Abstract
Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.
Original language | English |
---|---|
Title of host publication | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
Editors | Toon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo |
Publisher | Springer Verlag |
Pages | 386-403 |
Number of pages | 18 |
Volume | 8725 LNAI |
Edition | PART 2 |
ISBN (Print) | 9783662448502 |
DOIs | |
Publication status | Published - 1 Jan 2014 |
Event | European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France Duration: 15 Sept 2014 → 19 Sept 2014 |
Publication series
Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|
Number | PART 2 |
Volume | 8725 LNAI |
ISSN (Print) | 03029743 |
ISSN (Electronic) | 16113349 |
Conference
Conference | European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 |
---|---|
Country/Territory | France |
City | Nancy |
Period | 15/09/14 → 19/09/14 |
Research Groups and Themes
- Jean Golding
Fingerprint
Dive into the research topics of 'Rate-constrained ranking and the rate-weighted AUC'. Together they form a unique fingerprint.Projects
- 5 Finished
-
-
IEU Theme 2
Flach, P. A. (Principal Investigator), Gaunt, T. R. (Principal Investigator) & Gaunt, T. R. (Principal Investigator)
1/06/13 → 31/03/18
Project: Research
-
IEU Theme 3
Windmeijer, F. (Principal Investigator), Tilling, K. M. (Researcher) & Tilling, K. M. (Principal Investigator)
1/06/13 → 31/03/18
Project: Research