Distribution Calibration for Regression

Hao Song; Tom Diethe; Meelis Kull; Peter Flach

Distribution Calibration for Regression

Hao Song, Tom Diethe, Meelis Kull, Peter Flach

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

96 Downloads (Pure)

Abstract

We are concerned with obtaining well-calibrated output distributions from regression models. Such distributions allow us to quantify the uncertainty that the model has regarding the predicted target value. We introduce the novel concept of distribution calibration, and demonstrate its advantages over the existing definition of quantile calibration. We further propose a post-hoc approach to improving the predictions from previously trained regression models, using multi-output Gaussian Processes with a novel Beta link function. The proposed method is experimentally verified on a set of common regression models and shows improvements for both distribution-level and quantile-level calibration.

Original language	English
Title of host publication	International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA
Editors	Kamalika Chaudhuri, Ruslan Salakhutdinov
Publisher	Proceedings of Machine Learning Research
Pages	5897-5906
Number of pages	10
Publication status	Published - 15 May 2019

Publication series

Name	Proceedings of Machine Learning Research
Volume	97
ISSN (Print)	2640-3498

Structured keywords

Digital Health
SPHERE

Keywords

stat.ML
cs.AI
cs.LG

Access to Document

Full-text PDF (accepted author manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via PMLR at http://proceedings.mlr.press/v97/song19a.html . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 1.92 MB

http://proceedings.mlr.press/v97/song19a.html

Handle.net

Persistent link

SPHERE2
Craddock, I. J., Mirmehdi, M., Piechocki, R. J., Flach, P. A., Oikonomou, G., Burghardt, T., Damen, D., Santos-Rodriguez, R., O'Kane, A. A., McConville, R., Masullo, A. & Gooberman-Hill, R.
1/10/18 → 31/01/23
Project: Research, Parent

Classifier Calibration
Peter A Flach (Organiser), Miquel Perello Nieto (Organiser), Hao Song (Organiser), Meelis Kull (Organiser) & Telmo De Menezes E Silva Filho (Organiser)
14 Sep 2020
Activity: Participating in or organising an event types › Participation in workshop, seminar, course

Cite this

Song, H., Diethe, T., Kull, M., & Flach, P. (2019). Distribution Calibration for Regression. In K. Chaudhuri, & R. Salakhutdinov (Eds.), International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA (pp. 5897-5906). (Proceedings of Machine Learning Research; Vol. 97). Proceedings of Machine Learning Research. http://proceedings.mlr.press/v97/song19a.html

@inproceedings{89d18ff34fbb4cd5b018fa6a6c837806,

title = "Distribution Calibration for Regression",

abstract = " We are concerned with obtaining well-calibrated output distributions from regression models. Such distributions allow us to quantify the uncertainty that the model has regarding the predicted target value. We introduce the novel concept of distribution calibration, and demonstrate its advantages over the existing definition of quantile calibration. We further propose a post-hoc approach to improving the predictions from previously trained regression models, using multi-output Gaussian Processes with a novel Beta link function. The proposed method is experimentally verified on a set of common regression models and shows improvements for both distribution-level and quantile-level calibration. ",

keywords = "stat.ML, cs.AI, cs.LG",

author = "Hao Song and Tom Diethe and Meelis Kull and Peter Flach",

year = "2019",

month = may,

day = "15",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research",

pages = "5897--5906",

editor = "Kamalika Chaudhuri and Ruslan Salakhutdinov",

booktitle = "International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA",

}

Song, H, Diethe, T, Kull, M & Flach, P 2019, Distribution Calibration for Regression. in K Chaudhuri & R Salakhutdinov (eds), International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA. Proceedings of Machine Learning Research, vol. 97, Proceedings of Machine Learning Research, pp. 5897-5906. <http://proceedings.mlr.press/v97/song19a.html>

Distribution Calibration for Regression. / Song, Hao; Diethe, Tom; Kull, Meelis et al.

International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA. ed. / Kamalika Chaudhuri; Ruslan Salakhutdinov. Proceedings of Machine Learning Research, 2019. p. 5897-5906 (Proceedings of Machine Learning Research; Vol. 97).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Distribution Calibration for Regression

AU - Song, Hao

AU - Diethe, Tom

AU - Kull, Meelis

AU - Flach, Peter

PY - 2019/5/15

Y1 - 2019/5/15

N2 - We are concerned with obtaining well-calibrated output distributions from regression models. Such distributions allow us to quantify the uncertainty that the model has regarding the predicted target value. We introduce the novel concept of distribution calibration, and demonstrate its advantages over the existing definition of quantile calibration. We further propose a post-hoc approach to improving the predictions from previously trained regression models, using multi-output Gaussian Processes with a novel Beta link function. The proposed method is experimentally verified on a set of common regression models and shows improvements for both distribution-level and quantile-level calibration.

AB - We are concerned with obtaining well-calibrated output distributions from regression models. Such distributions allow us to quantify the uncertainty that the model has regarding the predicted target value. We introduce the novel concept of distribution calibration, and demonstrate its advantages over the existing definition of quantile calibration. We further propose a post-hoc approach to improving the predictions from previously trained regression models, using multi-output Gaussian Processes with a novel Beta link function. The proposed method is experimentally verified on a set of common regression models and shows improvements for both distribution-level and quantile-level calibration.

KW - stat.ML

KW - cs.AI

KW - cs.LG

M3 - Conference Contribution (Conference Proceeding)

T3 - Proceedings of Machine Learning Research

SP - 5897

EP - 5906

BT - International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA

A2 - Chaudhuri, Kamalika

A2 - Salakhutdinov, Ruslan

PB - Proceedings of Machine Learning Research

ER -

Distribution Calibration for Regression

Abstract

Publication series

Structured keywords

Keywords

Access to Document

Handle.net

Fingerprint

Projects

SPHERE2

Activities

Classifier Calibration

Cite this