Projects per year
Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.
|Title of host publication||Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|
|Editors||Toon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo|
|Number of pages||18|
|Publication status||Published - 1 Jan 2014|
|Event||European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France|
Duration: 15 Sep 2014 → 19 Sep 2014
|Name||Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|
|Conference||European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014|
|Period||15/09/14 → 19/09/14|
- Jean Golding
FingerprintDive into the research topics of 'Rate-constrained ranking and the rate-weighted AUC'. Together they form a unique fingerprint.
- 5 Finished
1/04/14 → 31/03/19
IEU Theme 3
Windmeijer, F., Tilling, K. M. & Tilling, K. M.
1/06/13 → 31/03/18
IEU Theme 2
Flach, P. A., Gaunt, T. R. & Gaunt, T. R.
1/06/13 → 31/03/18