Learning and Optimisation of Representations in Deep Learning

Abstract

We seek to explore the importance of representation learning in deep models like neural networks, with a particular emphasis on the learned function, rather than the weight matrices. Recent work (Deep Kernel Machines) studies representation learning by modifying the infinite-width limit in Neural Network Gaussian Process models, which don’t perform representation learning, so that their intermediate kernel matrices become learnable parameters, hence augmenting them with representation learning. However, this work only studied very simple regression models with a handful of input features. Deep learning’s success is most notable in domains such as image recognition, extracting meaningful features from high-dimensional unstructured data. Hence, in this thesis, we introduce convolutional deep kernel machines, which requires us to develop a sophisticated inducing point approximation scheme to allow tractable computation, and a number of other techniques for regularisation and numerical stability. The resulting model is a deep kernel method with learned Gram matrices, which achieves 94.5% test accuracy on the CIFAR-10 image classification dataset, matching a comparable ResNet20 neural network, and setting a new state-of-the-art for kernel methods. In addition to our work on convolutional deep kernel machines, we also develop a framework for efficiently estimating the contribution of different layers of a neural network to the learned representations at the output layer (i.e. the learned function), and utilise this framework to enable more efficient tuning of the learning rate hyperparameters in very large neural networks, using a scheme we call "Function-space Learning Rate Matching" (FLeRM).

Date of Award	20 Jan 2026
Original language	English
Awarding Institution	University of Bristol
Supervisor	Laurence Aitchison (Supervisor)