Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Toby J Perrett; Alessandro Masullo; Tilo Burghardt; Majid Mirmehdi; Dima  Damen

doi:10.1109/cvpr46437.2021.00054

Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Toby J Perrett, Alessandro Masullo, Tilo Burghardt, Majid Mirmehdi, Dima Damen

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

192 Citations (Scopus)

208 Downloads (Pure)

Abstract

We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.1Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

Original language	English
Title of host publication	2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	475-484
Number of pages	10
ISBN (Electronic)	978-1-6654-4509-2
ISBN (Print)	978-1-6654-4510-8
DOIs	https://doi.org/10.1109/cvpr46437.2021.00054
Publication status	Published - 2 Nov 2021
Event	Computer Vision and Pattern Recognition 2021 - Online Duration: 19 Jun 2021 → 25 Jun 2021 http://cvpr2021.thecvf.com/

Publication series

Name	Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	IEEE
ISSN (Print)	1063-6919
ISSN (Electronic)	2575-7075

Conference

Conference	Computer Vision and Pattern Recognition 2021
Abbreviated title	CVPR
Period	19/06/21 → 25/06/21
Internet address	http://cvpr2021.thecvf.com/

Research Groups and Themes

SPHERE
Digital Health

Access to Document

10.1109/cvpr46437.2021.00054Licence: Unspecified

Full-text PDF (author’s accepted manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via [insert publisher name] at [insert hyperlink] . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 6.39 MB

https://arxiv.org/abs/2101.06184Licence: CC BY

Handle.net

Persistent link

UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

Perrett, T. J., Masullo, A., Burghardt, T., Mirmehdi, M., & Damen, D. (2021). Temporal-Relational CrossTransformers for Few-Shot Action Recognition. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 475-484). (Conference on Computer Vision and Pattern Recognition (CVPR)). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/cvpr46437.2021.00054

@inproceedings{0e50eb4f330b4eb4af4996c3edb8a1b4,

title = "Temporal-Relational CrossTransformers for Few-Shot Action Recognition",

abstract = "We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.1Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12\%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.",

author = "Perrett, \{Toby J\} and Alessandro Masullo and Tilo Burghardt and Majid Mirmehdi and Dima Damen",

year = "2021",

month = nov,

day = "2",

doi = "10.1109/cvpr46437.2021.00054",

language = "English",

isbn = "978-1-6654-4510-8",

series = "Conference on Computer Vision and Pattern Recognition (CVPR)",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "475--484",

booktitle = "2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

address = "United States",

note = "Computer Vision and Pattern Recognition 2021, CVPR ; Conference date: 19-06-2021 Through 25-06-2021",

url = "http://cvpr2021.thecvf.com/",

}

Perrett, TJ, Masullo, A, Burghardt, T , Mirmehdi, M & Damen, D 2021, Temporal-Relational CrossTransformers for Few-Shot Action Recognition. in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Conference on Computer Vision and Pattern Recognition (CVPR), Institute of Electrical and Electronics Engineers (IEEE), pp. 475-484, Computer Vision and Pattern Recognition 2021, 19/06/21. https://doi.org/10.1109/cvpr46437.2021.00054

Temporal-Relational CrossTransformers for Few-Shot Action Recognition. / Perrett, Toby J; Masullo, Alessandro; Burghardt, Tilo et al.
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE), 2021. p. 475-484 (Conference on Computer Vision and Pattern Recognition (CVPR)).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Temporal-Relational CrossTransformers for Few-Shot Action Recognition

AU - Perrett, Toby J

AU - Masullo, Alessandro

AU - Burghardt, Tilo

AU - Mirmehdi, Majid

AU - Damen, Dima

PY - 2021/11/2

Y1 - 2021/11/2

N2 - We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.1Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

AB - We propose a novel approach to few-shot action recognition, finding temporally-corresponding frame tuples between the query and videos in the support set. Distinct from previous few-shot works, we construct class prototypes using the CrossTransformer attention mechanism to observe relevant sub-sequences of all support videos, rather than using class averages or single best matches. Video representations are formed from ordered tuples of varying numbers of frames, which allows sub-sequences of actions at different speeds and temporal offsets to be compared.1Our proposed Temporal-Relational CrossTransformers (TRX) achieve state-of-the-art results on few-shot splits of Kinetics, Something-Something V2 (SSv2), HMDB51 and UCF101. Importantly, our method outperforms prior work on SSv2 by a wide margin (12%) due to the its ability to model temporal relations. A detailed ablation showcases the importance of matching to multiple support set videos and learning higher-order relational CrossTransformers.

U2 - 10.1109/cvpr46437.2021.00054

DO - 10.1109/cvpr46437.2021.00054

M3 - Conference Contribution (Conference Proceeding)

SN - 978-1-6654-4510-8

T3 - Conference on Computer Vision and Pattern Recognition (CVPR)

SP - 475

EP - 484

BT - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - Computer Vision and Pattern Recognition 2021

Y2 - 19 June 2021 through 25 June 2021

ER -

Perrett TJ, Masullo A, Burghardt T , Mirmehdi M , Damen D. Temporal-Relational CrossTransformers for Few-Shot Action Recognition. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE). 2021. p. 475-484. (Conference on Computer Vision and Pattern Recognition (CVPR)). doi: 10.1109/cvpr46437.2021.00054

Temporal-Relational CrossTransformers for Few-Shot Action Recognition

Abstract

Publication series

Conference

Research Groups and Themes

Access to Document

Handle.net

Fingerprint

Projects

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this