Facilitating Bayesian Continual Learning by Natural Gradients and Stein Gradients

Yu Chen, Tom Diethe, Neil Lawrence

Research output: Contribution to conferenceConference Paper

30 Downloads (Pure)

Abstract

Continual learning aims to enable machine learning models to learn a general solution space for past and future tasks in a sequential manner. Conventional models tend to forget the knowledge of previous tasks while learning a new task, a phenomenon known as catastrophic forgetting. When using Bayesian models in continual learning, knowledge from previous tasks can be retained in two ways: 1). posterior distributions over the parameters, containing the knowledge gained from inference in previous tasks, which then serve as the priors for the following task; 2). coresets, containing knowledge of data distributions of previous tasks. Here, we show that Bayesian continual learning can be facilitated in terms of these two means through the use of natural gradients and Stein gradients respectively.
Original languageEnglish
Number of pages8
Publication statusPublished - 24 Apr 2019
EventContinual Learning Workshop of 32nd Conference on Neural Information Processing Systems (NeurIPS 2018) - Montreal, Quebec, Canada
Duration: 2 Dec 20188 Dec 2018
Conference number: 32
https://nips.cc/Conferences/2018

Workshop

WorkshopContinual Learning Workshop of 32nd Conference on Neural Information Processing Systems (NeurIPS 2018)
Abbreviated titleNeurIPS 2018
CountryCanada
CityMontreal, Quebec
Period2/12/188/12/18
Internet address

Bibliographical note

Presented at the NeurIPS 2018 workshop on Continual Learning https://sites.google.com/view/continual2018/home

Keywords

  • cs.LG
  • cs.AI
  • stat.ML

Fingerprint Dive into the research topics of 'Facilitating Bayesian Continual Learning by Natural Gradients and Stein Gradients'. Together they form a unique fingerprint.

Cite this