The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

50 Citations (Scopus)
97 Downloads (Pure)


We present a new model to determine relative skill from long videos, through learnable temporal attention modules. Skill determination is formulated as a ranking problem, making it suitable for common and generic tasks. However, for long videos, parts of the video are irrelevant for assessing skill, and there may be variability in the skill exhibited throughout a video. We therefore propose a method which assesses the relative overall level of skill in a long video by attending to its skill-relevant parts. Our approach trains temporal attention modules, learned with only video-level supervision, using a novel rank-aware loss function. In addition to attending to task relevant video parts, our proposed loss jointly trains two attention modules to separately attend to video parts which are indicative of higher (pros) and lower (cons) skill. We evaluate our approach on the EPIC-Skills dataset and additionally annotate a larger dataset from YouTube videos for skill determination with five previously unexplored tasks. Our method outperforms previous approaches and classic softmax attention on both datasets by over 4% pairwise accuracy, and as much as 12% on individual tasks. We also demonstrate our model’s ability to attend to rank-aware parts of the video.
Original languageEnglish
Title of host publication2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
ISBN (Print)978-1-7281-3293-8
Publication statusPublished - 11 Mar 2019
EventComputer Vision and Pattern Recognition (CVPR) - Rhode Island, United States
Duration: 16 Jun 201221 Jun 2012

Publication series

ISSN (Print)1063-6919
ISSN (Electronic)2575-7075


ConferenceComputer Vision and Pattern Recognition (CVPR)
Country/TerritoryUnited States
CityRhode Island

Bibliographical note

Proxy date of acceptance added to output record.


Dive into the research topics of 'The Pros and Cons: Rank-aware Temporal Attention for Skill Determination in Long Videos'. Together they form a unique fingerprint.

Cite this