Reinventing Machine Learning with ROC Analysis (invited talk)

Receiver Operating Characteristics (ROC) Analysis originated from signal detection theory, as a model of how well a receiver is able to detect a signal in the presence of noise. Its key feature is the distinction between hit rate (or true positive rate) and false alarm rate (or false positive rate) as two separate performance measures. It has been introduced to machine learning relatively recently, in response to classification tasks with skewed class distributions or misclassification costs. ROC analysis is set to cause a paradigm shift in classification-oriented machine learning. Separating performance on classes can help us understand the behaviour and skew-sensitivity of many machine learning metrics by plotting their isometrics in ROC space; develop methods and algorithms to improve the Area Under the ROC Curve (AUC) of a model; understand fundamental classifier training algorithms by visualising how they split up ROC space or its unnormalised cousin, PN space. In this talk I will mainly illustrate the latter point.