Gendered Neural Networks

: Representation and generalisation of grammatical gender in language models

Abstract

Neural network language models are remarkable at mimicking the complexities of natural language. Their increasing success in practical natural language processing tasks has, however, come with a trade-off in the mechanistic understanding and interpretability of their linguistic capabilities. Do language models really develop human-like linguistic knowledge?
One fundamental idea that supports generalisation in humans, is the ability to form abstract grammatical categories and grammatical agreement rules. Grammatical gender is one such abstract category, that is an inherent property of nouns which cannot always be deduced from semantic contexts, making it particularly challenging. In three studies, we evaluate whether language models develop an abstract representation of grammatical gender, like humans.
We show that transformer and LSTM language models, particularly LSTMs, effectively handle grammatical gender agreement in French, even in complex long-distance constructions with attractors. We demonstrate that this ability is supported by a highly sparse encoding mechanism in LSTMs, with distinct gender units that abstractly encode grammatical gender across nouns and agreement configurations. Next, in a few-shot learning study, we show that the models exhibit proficiency in learning and generalising gender categories of novel nouns across different agreement contexts. Models achieve this purely based on updates to the representational layers, suggesting that grammatical gender is abstractly encoded in word embeddings. However, few-shot learning varies with gender agreement configurations. Our studies also reveal asymmetric agreement performance and encoding mechanisms across noun gender and number categories, pointing to a ‘default masculine reasoning’ strategy, which we also observed in our human word-learning experiments.
While these findings provide some evidence for the abstract encoding and generalisation of grammatical gender in language models, the difference in gender agreement performance, encoding mechanisms in LSTMs, and few-shot learning behaviours across syntactic configurations highlight the complexity and variability in how grammatical gender is employed during agreement. This invites further exploration of the gender agreement mechanisms in language models.

Date of Award	24 Apr 2024
Original language	English
Awarding Institution	University of Bristol
Supervisor	Conor Houghton (Supervisor) & Nina Kazanina (Supervisor)