Deep Convolutional Networks as shallow Gaussian Processes

Adria Garriga-Alonso; Carl E. Rasmussen; Laurence Aitchison

Deep Convolutional Networks as shallow Gaussian Processes

Adria Garriga-Alonso
, Carl E. Rasmussen
, Laurence Aitchison

Bristol Neuroscience

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Abstract

We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.

Original language	English
Title of host publication	International Conference on Learning Representations
Pages	1-16
Publication status	Published - 6 May 2019

Publication series

Name	ICLR

Access to Document

https://arxiv.org/abs/1808.05587Licence: CC BY
https://openreview.net/forum?id=Bklfsi0cKmLicence: Unspecified

Handle.net

Persistent link

Dr Laurence Aitchison
- laurence.aitchisonbristol.acuk
- School of Engineering Mathematics and Technology - Senior Lecturer
- Bristol Neuroscience
Person: Academic , Member

Cite this

@inproceedings{3e20975001ce4a26b8a50075b81c4ed0,

title = "Deep Convolutional Networks as shallow Gaussian Processes",

abstract = "We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike {"}deep kernels{"}, has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84\% classification error on MNIST, a new record for GP with a comparable number of parameters.",

author = "Adria Garriga-Alonso and Rasmussen, \{Carl E.\} and Laurence Aitchison",

year = "2019",

month = may,

day = "6",

language = "English",

series = "ICLR",

pages = "1--16",

booktitle = "International Conference on Learning Representations",

}

TY - GEN

T1 - Deep Convolutional Networks as shallow Gaussian Processes

AU - Garriga-Alonso, Adria

AU - Rasmussen, Carl E.

AU - Aitchison, Laurence

PY - 2019/5/6

Y1 - 2019/5/6

N2 - We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.

AB - We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.

M3 - Conference Contribution (Conference Proceeding)

T3 - ICLR

SP - 1

EP - 16

BT - International Conference on Learning Representations

ER -

Deep Convolutional Networks as shallow Gaussian Processes

Abstract

Publication series

Access to Document

Handle.net

Fingerprint

Profiles

Dr Laurence Aitchison

Cite this