Unsupervised domain adaptation for fine-grained action understanding

Student thesis: Doctoral ThesisDoctor of Philosophy (PhD)

Abstract

Fine-grained actions are short actions that typically last a matter of seconds. However, due to the difficulty in collecting and annotating fine-grained actions, datasets contain limited variability in their collection. Many datasets are collected in a single environment, with few participants and cameras. Computer vision models trained on such datasets may not perform well on videos encountered when they are deployed. This work showcases several domains shifts in the large scale dataset of fine-grained actions, EPIC-KITCHENS.

This thesis focuses on unsupervised domain adaptation for fine-grained action understanding. This assumes there is a domain shift between the labelled videos used for training (the source domain) and videos used for testing (the target domain). With access to unlabelled videos from the target domain, the aim is to improve the performance of fine-grained action understanding tasks. Unsupervised domain adaptation reduces the high cost of annotating fine-grained actions, which is often expensive or impractical in the target domain.

Videos depicting fine-grained actions contain both visual and motion information, as well as audio and often textual descriptions. This work explores utilising these multiple modalities to improve domain adaptation, as well as learn a representation of fine-grained actions. Some modalities will be more robust than others to different domain shifts, for example motion is more robust than RGB to environmental changes. A domain adaptation solution is proposed which improves action recognition performance by exploiting the differing level of robustness of video modalities to do main shifts. Additionally, cross-modal tasks can be used to learn discriminative information about fine-grained actions. A domain adaptation solution is proposed to adapt a text-to-video retrieval system to a novel set of uncaptioned videos.
Date of Award25 Jan 2022
Original languageEnglish
Awarding Institution
  • University of Bristol
SupervisorDima Damen (Supervisor) & Walterio W Mayol-Cuevas (Supervisor)

Cite this

'