Using Score-based Methods for Unnormalisable Probability Density Estimation

: Truncated Density Estimation and Parameter Derivative Estimation

Abstract

Classical statistical modelling such as maximum likelihood estimation relies on knowledge of the normalising constant of a probability density model. Under certain cases, for example where data are observed on a generic truncated domain, the normalising constant is intractable. Whilst conventional methods usually approximate this term via numerical integration, methods such as score matching (Hyvärinen, 2005) and minimum Stein discrepancy estimators (Barp et al., 2019; Liu et al., 2016) bypass its evaluation entirely.

In chapter 3, we present a method that uses score matching to define a density estimator on a generic truncation domain. This method relies on a weighting function that is equal to zero at the boundary of the domain, for which we propose a distance function which arises naturally from maximising a Stein discrepancy. Numerical experiments show the potential of the proposed
method across a range of experiments, including real world Chicago crime data, and correcting the over-compensation from outlier trimming.

In chapter 4, we extend the work from chapter 3 where data are both truncated and lie on the surface of a manifold. We present an application of this method to the sphere, and propose two distance functions as the corresponding weighting function. Experiments on the sphere show that this method achieves lower estimation error than prior Euclidean domain methods. We present a real-world application estimating the mean of storm locations truncated over the continental United States.

In chapter 5, we define an approximate Stein class for which the Stein identity holds approximately. This enables construction of a truncated kernelised Stein discrepancy, which only requires access to a set of boundary points. In experiments, we show that even with this relaxed set of assumptions, the method is comparable in performance to previous methods.

Finally, in chapter 6, we derive an estimator of the time-varying derivative of the parameter of an unnormalised exponential family density using time score matching (Choi et al., 2022). We present a changepoint detection method based on the asymptotic variance of this estimator which needs no prior assumptions on the type of changepoint expected. This highly flexible method provides comparable results to previous popular changepoint detection methods. We show its applicability to real-world applications, as well as the ability to detect when out-of-distribution images are introduced to a dataset using a neural network representation.

Date of Award	1 Oct 2024
Original language	English
Awarding Institution	University of Bristol
Supervisor	Song Liu (Supervisor) & Oliver T Johnson (Supervisor)