Abstract
Fields work in very different ways - but fail in similar ways. This thesis covers some of my work in epidemiology, psychology, and machine learning with the common thread of shared methodological issues. I identify frameworks which fail to cover actual practice, practices which fail to live up to normative principles, and propose practices which are sometimes able to address some failures at some cost.In epidemiology I take a Bayesian approach to infectious disease modelling and infer the effect of entire populations wearing face masks during the Covid pandemic – with the key caveat that this is an observational study. I identify a ubiquitous methodological mistake (using mandate timings as a proxy for wearing behaviour, when these are, surprisingly, not strongly correlated).
In psychology I synthesise theories of the replication crisis and report on initiating a large (n = 1931) dataset of replication studies covering the original effect sizes, replication effect sizes, and both raw and recalculated statistics. These nonrandom data still give some insight into post-crisis psychology, confirming past results showing that even the sign of effects is often not replicable. I note a pattern of considerable ‘shrinkage’ in effect sizes between the original study and their replications.
In machine learning I trace recent methodological changes and provide a novel analysis of roughly forty ways that ML evaluations are often misleading. Each chapter contains a self-contained background section for its respective field. I conclude with lessons for each field from the other two.
| Date of Award | 10 Dec 2024 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | Laurence Aitchison (Supervisor), Nathan F Lepora (Supervisor) & Martha Lewis (Supervisor) |