Reinforcement learning models of hippocampal and cerebellar networks

Abstract

Animals and humans possess a remarkable decision-making capacity to navigate intricate environments with ease. The ability to explore efficiently is an essential skill for survival, including in situations with limited visibility. However, the neural mechanisms behind such a process are yet to be fully understood. This thesis aims to reduce this gap between nature and our understanding by proposing a computational reinforcement learning (RL) model of the hippocampus that is trained to perform reward-based navigational tasks in partially observable environments. The findings reveal that agents with recurrent hippocampal networks outperform pure feedforward networks, closely mimicking animal behaviour in allocentric and egocentric tasks. Additionally, dimensionality reduction methods are utilised to support that our models accurately predict reward, strategies, and temporal activity features, which we validated against large-scale hippocampal neuronal recordings. Moreover, the RL agents display state-specific trajectories and action certainty, which are in line with empirical observations. Interestingly, agents trained in fully observable settings fail to reproduce experimental data, highlighting the importance of partial observability in understanding goal-oriented behaviour. Finally, our hippocampal-like agents are able to generalise across new task conditions, supporting the hippocampal network's role in learning within naturalistic environments.

Recent evidence suggests that the cerebellum controls hippocampal networks during navigation. To study the potential role of cerebellar networks in the hippocampal-dependent navigation, we built on our hippocampal RL model to introduce a system-level RL model of cerebello-hippocampal networks. In this model, the cerebellum drives CA3 hippocampal networks with optimised task-specific inputs. Our results show that predictive input from the cerebellum leads to accelerated and more stable learning in models with non-plastic CA3 networks, supporting experimental findings of impaired navigational performance in animals with cerebellar deficits. Our model also successfully demonstrates the emergence of hippocampal spatial features, such as place and border cells, similar to those observed in the rat's hippocampus. Analysis of neuronal dynamics further supports the hypothesis that models without a cerebellum show less stable hippocampal representations, aligning with recent experimental studies in spatial navigation. In summary, our findings suggest that the cerebellum drives the hippocampus with task-specific information. This dynamic interaction improves goal-oriented learning, leading to more optimal trajectories that are similar to those observed in animal behaviour.

Overall, this thesis highlights the computational principles by which the hippocampus and cerebellum support the brain for navigation under naturalistic conditions.

Date of Award	18 Jun 2024
Original language	English
Awarding Institution	University of Bristol
Supervisor	Rui Ponte Costa (Supervisor) & Nathan F Lepora (Supervisor)