Generalized Additive Models for Gigadata: Modeling the U.K. Black Smoke Network Daily Data

We develop scalable methods for fitting penalized regression spline based generalized additive models with of the order of 10^4 coefficients to up to 10^8 data. Computational feasibility rests on: (i) a new iteration scheme for estimation of model coefficients and smoothing parameters, avoiding poorly scaling matrix operations; (ii) parallelization of the iteration's pivoted block Cholesky and basic matrix operations; (iii) the marginal discretization of model covariates to reduce memory footprint, with efficient scalable methods for computing required crossproducts directly from the discrete representation. Marginal discretization enables much finer discretization than joint discretization would permit. We were motivated by the need to model four decades worth of daily particulate data from the UK Black Smoke and Sulphur Dioxide monitoring network. Although reduced in size recently, over 2000 stations have at some time been part of the network, resulting in some 10 million measurements. Modelling at a daily scale is desirable for accurate trend estimation and mapping, and to provide daily exposure estimates for epidemiological cohort studies. Because of the data set size, previous work has focused on modelling time or space averages pollution levels, but this is unsatisfactory from a health perspective, since it is often acute exposure locally and on the time scale of days that is of most importance in driving adverse health outcomes.
If computed by conventional means our black smoke model would require a half terabyte of storage just for the model matrix, whereas we are able to compute with it on a desktop workstation. The best previously available reduced memory footprint method would have required three orders of magnitude more computing time than our new method.