Rate-constrained ranking and the rate-weighted AUC

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

4 Citations (Scopus)

Abstract

Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model's performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology. © 2014 Springer-Verlag.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
EditorsToon Calders, Floriana Esposito, Eyke Hullermeier, Rosa Meo
PublisherSpringer Verlag
Pages386-403
Number of pages18
Volume8725 LNAI
EditionPART 2
ISBN (Print)9783662448502
DOIs
Publication statusPublished - 1 Jan 2014
EventEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014 - Nancy, France
Duration: 15 Sept 201419 Sept 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 2
Volume8725 LNAI
ISSN (Print)03029743
ISSN (Electronic)16113349

Conference

ConferenceEuropean Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2014
Country/TerritoryFrance
CityNancy
Period15/09/1419/09/14

Research Groups and Themes

  • Jean Golding

Fingerprint

Dive into the research topics of 'Rate-constrained ranking and the rate-weighted AUC'. Together they form a unique fingerprint.
  • ConDuCT-II

    Blazeby, J. (Principal Investigator)

    1/04/1431/03/19

    Project: Research

  • IEU Theme 2

    Flach, P. A. (Principal Investigator), Gaunt, T. R. (Principal Investigator) & Gaunt, T. R. (Principal Investigator)

    1/06/1331/03/18

    Project: Research

  • IEU Theme 3

    Windmeijer, F. (Principal Investigator), Tilling, K. M. (Researcher) & Tilling, K. M. (Principal Investigator)

    1/06/1331/03/18

    Project: Research

Cite this