Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Toby J Perrett; Alessandro Masullo; Tilo Burghardt; Majid Mirmehdi; Dima  Damen

doi:10.1109/cvpr46437.2021.00054

Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Toby J Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, Dima Damen

Research output: Contribution to conference › Conference Paper › peer-review

172 Citations (Scopus)

184 Downloads (Pure)

Abstract

We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.

Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

Original language	English
Number of pages	8
DOIs	https://doi.org/10.1109/cvpr46437.2021.00054
Publication status	Published - 25 Jun 2021
Event	Computer Vision and Pattern Recognition 2021 - Online Duration: 19 Jun 2021 → 25 Jun 2021 http://cvpr2021.thecvf.com/

Conference

Conference	Computer Vision and Pattern Recognition 2021
Abbreviated title	CVPR
Period	19/06/21 → 25/06/21
Internet address	http://cvpr2021.thecvf.com/

Research Groups and Themes

SPHERE
Digital Health

Access to Document

10.1109/cvpr46437.2021.00054

Full-text PDF (author’s accepted manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via [insert publisher name] at [insert hyperlink] . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 6.39 MB

Handle.net

Persistent link

UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

@conference{0e50eb4f330b4eb4af4996c3edb8a1b4,

title = "Temporal-Relational CrossTransformers for Few-Shot Action Recognition",

abstract = "We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.",

author = "Perrett, {Toby J} and Alessandro Masullo and Tilo Burghardt and Majid Mirmehdi and Dima Damen",

year = "2021",

month = jun,

day = "25",

doi = "10.1109/cvpr46437.2021.00054",

language = "English",

note = "Computer Vision and Pattern Recognition 2021, CVPR ; Conference date: 19-06-2021 Through 25-06-2021",

url = "http://cvpr2021.thecvf.com/",

}

TY - CONF

T1 - Temporal-Relational CrossTransformers for Few-Shot Action Recognition

AU - Perrett, Toby J

AU - Masullo, Alessandro

AU - Burghardt, Tilo

AU - Mirmehdi, Majid

AU - Damen, Dima

PY - 2021/6/25

Y1 - 2021/6/25

N2 - We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

AB - We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

U2 - 10.1109/cvpr46437.2021.00054

DO - 10.1109/cvpr46437.2021.00054

M3 - Conference Paper

T2 - Computer Vision and Pattern Recognition 2021

Y2 - 19 June 2021 through 25 June 2021

ER -

Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Abstract

Conference

Research Groups and Themes

Access to Document

Handle.net

Fingerprint

Projects

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this