Processing questions with multi-task sentence embedding

Abstract

Deep neural networks were widely applied to understanding human language in the past decade. In order to let the machine build a better comprehension of text, researchers created various tasks to train the algorithms and language models. The machine was trained to understand articles, emails, and even shorter text like Tweets. With the development of online questioning communities and voice assistants, we notice a new type of text data coming into natural language processing research: Questions. Analysing questions can provide a new perspective on understanding communities and people's interests. Furthermore, while raising different kinds of questions, people also seek similar questions asked by others and answers. Thousands of questions are asked on a daily basis. With this large amount of data collected, it is time-consuming to analyse all the questions by a human. It will be helpful if a model can automatically process the questions.
In this thesis, we focus on training and fine-tuning Bidirectional Encoder Representations from Transformers (BERT), one of the transformer-based neural networks that created leading performance in many language comprehension tasks as well as generating sentence representation in a high dimensional vector space. To understand how BERT projects the sentences into vectors, we investigate the composition of the sentence representations generated by BERT with a simple sentence corpus: each sentence comprises only a subject, verb, and object. We decompose the sentence embedding into its attributes: words. The attribute representations can be used to predict the representation of an unseen sentence.
We also train a question-specialised version of BERT (QBERT) to become a ``generalist'' for analysing the questions in three different tasks: question topic classification, equivalent question detection, and question answering. QBERT is trained as a ``generalist'' based on multi-task learning techniques. The idea of a ``generalist'' is that QBERT should deliver good performance on a range of tasks involving questions in natural language, trading off some accuracy for robust performance across the various tasks. An important challenge is if labelled data from one task can be beneficial in fine-tuning the internal representations of QBERT, also when used on a different task.
With one version of QBERT we trained, QBERT-RR, we propose a pipeline for processing questions and apply it to analyse a new question corpus collected by ``We The Curious'', a science centre located in Bristol. There are more than 10,000 questions collected. Researching these questions has the potential to help the museum understand the visitors' curiosity and create exhibitions or educational content that are more relevant to their interests and lives. On top of that, QBERT-RR also answers some of the questions with one-sentence answers using a large-scale external knowledge source.

Date of Award	9 May 2023
Original language	English
Awarding Institution	University of Bristol
Supervisor	Nello Cristianini (Supervisor), Martha Lewis (Supervisor) & Dima Damen (Supervisor)