Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing

Hao Song, Peter A Flach*

*Corresponding author for this work

Research output: Contribution to journalArticle (Academic Journal)peer-review

1 Citation (Scopus)


Progress in predictive machine learning is typically measured on the basis of performance comparisons on benchmark datasets. Traditionally these kinds of empirical evaluation are carried out on large numbers of datasets, but this is becoming increasingly hard due to computational requirements and the often large number of alternative methods to compare against. In this paper we investigate adaptive approaches to achieve better efficiency on model benchmarking. For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model’s goodness, from which to extrapolate to performance on other datasets. To this end, we adapt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests. We propose certain modifications following the requirements of machine learning experiments, and present experimental results to validate the approach.
Original languageEnglish
Pages (from-to)110-118
Number of pages9
JournalInternational Journal of Interactive Multimedia and Artificial Intelligence
Issue number5
Publication statusPublished - 23 Feb 2021

Bibliographical note

Publisher Copyright:
© 2021, Universidad Internacional de la Rioja. All rights reserved.

Structured keywords

  • Interactive AI


Dive into the research topics of 'Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing'. Together they form a unique fingerprint.

Cite this