The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction

Alexandros G Stergiou; Dima  Damen

doi:10.1109/CVPR52729.2023.01413

The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction

Alexandros G Stergiou^*, Dima Damen

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

8 Citations (Scopus)

33 Downloads (Pure)

Abstract

Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations. † † Code is available at: https://tinyurl.com/temprog

Original language	English
Title of host publication	2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	14709-14719
Number of pages	11
ISBN (Electronic)	9798350301298
ISBN (Print)	9798350301304
DOIs	https://doi.org/10.1109/CVPR52729.2023.01413
Publication status	Published - 22 Aug 2023
Event	IEEE/CVF Computer Vision and Pattern Recognition - Vancouver, Canada Duration: 18 Jun 2023 → 23 Jun 2023

Publication series

Name	IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	IEEE
ISSN (Print)	1063-6919
ISSN (Electronic)	2575-7075

Conference

Conference	IEEE/CVF Computer Vision and Pattern Recognition
Abbreviated title	CVPR
Country/Territory	Canada
City	Vancouver
Period	18/06/23 → 23/06/23

Access to Document

10.1109/CVPR52729.2023.01413Licence: Unspecified

Full-text PDF (accepted manuscript)
This is the accepted author manuscript (AAM). The final published version (Version of Record) can be found on the publisher's website. The copyright of any third-party content, such as images, remains with the copyright holder.
Accepted author manuscript, 8.72 MB

https://arxiv.org/abs/2204.13340Licence: CC BY

Handle.net

Persistent link

UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS-100
Aldamen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

Stergiou, A. G., & Damen, D. (2023). The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 14709-14719). (IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/CVPR52729.2023.01413

@inproceedings{d95661ba22ee4537b36966c05b292171,

title = "The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction",

abstract = "Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations. † † Code is available at: https://tinyurl.com/temprog",

author = "Stergiou, {Alexandros G} and Dima Damen",

year = "2023",

month = aug,

day = "22",

doi = "10.1109/CVPR52729.2023.01413",

language = "English",

isbn = "9798350301304",

series = "IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "14709--14719",

booktitle = "2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

address = "United States",

note = "IEEE/CVF Computer Vision and Pattern Recognition, CVPR ; Conference date: 18-06-2023 Through 23-06-2023",

}

Stergiou, AG & Damen, D 2023, The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction. in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Institute of Electrical and Electronics Engineers (IEEE), pp. 14709-14719, IEEE/CVF Computer Vision and Pattern Recognition, Vancouver, Canada, 18/06/23. https://doi.org/10.1109/CVPR52729.2023.01413

The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction. / Stergiou, Alexandros G; Damen, Dima .
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE), 2023. p. 14709-14719 (IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - The Wisdom of Crowds

T2 - IEEE/CVF Computer Vision and Pattern Recognition

AU - Stergiou, Alexandros G

AU - Damen, Dima

PY - 2023/8/22

Y1 - 2023/8/22

N2 - Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations. † † Code is available at: https://tinyurl.com/temprog

AB - Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these towers. Extensive experiments over four video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of encoder architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations. † † Code is available at: https://tinyurl.com/temprog

U2 - 10.1109/CVPR52729.2023.01413

DO - 10.1109/CVPR52729.2023.01413

M3 - Conference Contribution (Conference Proceeding)

SN - 9798350301304

T3 - IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

SP - 14709

EP - 14719

BT - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 18 June 2023 through 23 June 2023

ER -

The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction

Abstract

Publication series

Conference

Access to Document

Handle.net

Fingerprint

Projects

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Datasets

EPIC-KITCHENS-100

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this