Diversity and degrees of freedom in regression ensembles

Henry Reeve, Gavin Brown

Research output: Contribution to journalArticle (Academic Journal)

Abstract

Ensemble methods are a cornerstone of modern machine learning. The performance of an ensemble depends crucially upon the level of diversity between its constituent learners. This paper establishes a connection between diversity and degrees of freedom (i.e. the capacity of the model), showing that diversity may be viewed as a form of inverse regularisation. This is achieved by focusing on a previously published algorithm Negative Correlation Learning(NCL), in which model diversity is explicitly encouraged through a diversity penalty term in the loss function. We provide an exact formula for the effective degrees of freedom in an NCL ensemble with fixed basis functions, showingthat it is a continuous, convex and monotonically increasing function of the diversity parameter. We demonstrate a connection to Tikhonov regularisation and show that, with an appropriately chosen diversity parameter, an NCLensemble can always outperform the unregularised ensemble in the presence of noise. We demonstrate the practical utility of our approach by deriving a method to efficiently tune the diversity parameter. Finally, we use a Monte-Carloestimator to extend the connection between diversity and degrees of freedom to ensembles of deep neural networks.
Original languageEnglish
JournalNeurocomputing
DOIs
Publication statusPublished - 2018

Keywords

  • Degrees of Freedom
  • Negative correlation learning
  • Tikhonov regularisation
  • Ensembles
  • Stein's unbiased risk estimate
  • deep neural networks

Fingerprint Dive into the research topics of 'Diversity and degrees of freedom in regression ensembles'. Together they form a unique fingerprint.

Cite this