Generalized Additive Models for Gigadata: Modeling the U.K. Black Smoke Network Daily Data

Simon N Wood, Zheyuan Li, Gavin Shaddick, Nicole H. Augustin

Research output: Contribution to journalArticle (Academic Journal)

31 Citations (Scopus)
574 Downloads (Pure)

Abstract

We develop scalable methods for fitting penalized regression spline based generalized additive models with of the order of 10^4 coefficients to up to 10^8 data. Computational feasibility rests on: (i) a new iteration scheme for estimation of model coefficients and smoothing parameters, avoiding poorly scaling matrix operations; (ii) parallelization of the iteration's pivoted block Cholesky and basic matrix operations; (iii) the marginal discretization of model covariates to reduce memory footprint, with efficient scalable methods for computing required crossproducts directly from the discrete representation. Marginal discretization enables much finer discretization than joint discretization would permit. We were motivated by the need to model four decades worth of daily particulate data from the UK Black Smoke and Sulphur Dioxide monitoring network. Although reduced in size recently, over 2000 stations have at some time been part of the network, resulting in some 10 million measurements. Modelling at a daily scale is desirable for accurate trend estimation and mapping, and to provide daily exposure estimates for epidemiological cohort studies. Because of the data set size, previous work has focused on modelling time or space averages pollution levels, but this is unsatisfactory from a health perspective, since it is often acute exposure locally and on the time scale of days that is of most importance in driving adverse health outcomes.
If computed by conventional means our black smoke model would require a half terabyte of storage just for the model matrix, whereas we are able to compute with it on a desktop workstation. The best previously available reduced memory footprint method would have required three orders of magnitude more computing time than our new method.
Original languageEnglish
Pages (from-to)1199-1210
Number of pages12
JournalJournal of the American Statistical Association
Volume112
Issue number519
Early online date25 Apr 2017
DOIs
Publication statusPublished - 3 Jul 2017

Structured keywords

  • Jean Golding

Fingerprint Dive into the research topics of 'Generalized Additive Models for Gigadata: Modeling the U.K. Black Smoke Network Daily Data'. Together they form a unique fingerprint.

Cite this