Skip to content

Tracking and modelling prices using web-scraped price microdata: towards automated daily consumer price index forecasting

Research output: Contribution to journalArticle

  • Ben Powell
  • Guy Nason
  • Duncan Elliott
  • Matthew Mayhew
  • Jennifer Davies
  • Joe Winton
Original languageEnglish
Pages (from-to)737-756
Number of pages20
JournalJournal of the Royal Statistical Society: Series A
Issue number3
Early online date15 Sep 2017
DateAccepted/In press - 20 Jul 2017
DateE-pub ahead of print - 15 Sep 2017
DatePublished (current) - 1 Jun 2018


With the increasing relevance and availability of on-line prices that we see today, it is natural to ask whether the prediction of the consumer price index (CPI), or related statistics, may usefully be computed more frequently than existing monthly schedules allow for. The simple answer is ‘yes’, but there are challenges to be overcome first. A key challenge, addressed by our work, is that web-scraped price data are extremely messy and it is not obvious, a priori, how to reconcile them with standard CPI statistics. Our research focuses on average prices and disaggregated CPI at the level of product categories (lager, potatoes, etc.) and develops a new model that describes the joint time evolution of latent daily log-inflation rates driving prices seen on the Internet and prices recorded in official surveys, with the model adapting to various product categories. Our model reveals the differing levels of dynamic behaviour across product category and, correspondingly, differing levels of predictability. Our methodology enables good prediction of product-category-specific CPI immediately before their release. In due course, with increasingly complete web-scraped data, combined with the best survey data, the prospect of more frequent intermonth aggregated CPI prediction is an achievable goal.

    Research areas

  • Dynamic inflation model, High frequency inflation prediction, Inflation estimation, State space model

Download statistics

No data available



  • Full-text PDF (accepted author manuscript)

    Accepted author manuscript, 1.52 MB, PDF document

    Embargo ends: 1/01/00

    Request copy

  • Full-text PDF (final published version)

    Rights statement: This is the final published version of the article (version of record). It first appeared online via WILEY at Please refer to any applicable terms of use of the publisher.

    Final published version, 4.55 MB, PDF document

    Licence: CC BY


View research connections

Related faculties, schools or groups