Skip to content

EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

Research output: Contribution to conferencePaper

Original languageEnglish
Number of pages10
DateAccepted/In press - 22 Jul 2019
DatePublished (current) - 2 Nov 2019
EventIEEE/CVF International Conference on Computer Vision (ICCV) 2019 - Korea, Seoul
Duration: 27 Oct 20192 Nov 2019

Conference

ConferenceIEEE/CVF International Conference on Computer Vision (ICCV) 2019
CitySeoul
Period27/10/192/11/19

Abstract

We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multimodal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities – RGB, Flow and Audio – and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities. We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.

Event

IEEE/CVF International Conference on Computer Vision (ICCV) 2019

Duration27 Oct 20192 Nov 2019
Location of eventKorea
CitySeoul

Event: Conference

Download statistics

No data available

Documents

Documents

View research connections

Related faculties, schools or groups