Play Fair: Frame Attributions in Video Models

Will Price; Dima  Damen

doi:10.1007/978-3-030-69541-5_29

Play Fair: Frame Attributions in Video Models

Will Price, Dima Damen

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

15 Downloads (Pure)

Abstract

In this paper, we introduce an attribution method for explaining action recognition models. Such models fuse information from multiple frames within a video, through score aggregation or relational reasoning. We break down a model’s class score into the sum of contributions from each frame, fairly. Our method adapts an axiomatic solution to fair reward distribution in cooperative games, known as the Shapley value, for elements in a variable-length sequence, which we call the Element Shapley Value (ESV). Critically, we propose a tractable approximation of ESV that scales linearly with the number of frames in the sequence.

We employ ESV to explain two action recognition models (TRN and TSN) on the ﬁne-grained dataset Something-Something. We oﬀer detailed analysis of supporting/distracting frames, and the relationships of ESVs to the frame’s position, class prediction, and sequence length. We compare ESV to naive baselines and two commonly used feature attribution methods: Grad-CAM and Integrated-Gradients.

Original language	English
Title of host publication	Computer Vision - ACCV 2020
Subtitle of host publication	15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part V
Editors	Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, Jianbo Shi
Publisher	Springer
Pages	480-497
Number of pages	18
ISBN (Electronic)	9783030695415
ISBN (Print)	9783030695408
DOIs	https://doi.org/10.1007/978-3-030-69541-5_29
Publication status	Published - 25 Feb 2021
Event	15th Asian Conference on Computer Vision - Duration: 30 Nov 2020 → 4 Dec 2020 http://accv2020.kyoto/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer International Publishing
Volume	12626
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	15th Asian Conference on Computer Vision
Period	30/11/20 → 4/12/20
Internet address	http://accv2020.kyoto/

Access to Document

10.1007/978-3-030-69541-5_29

Full-text PDF (author’s accepted manuscript)
This is the accepted author manuscript (AAM) of the article which has been made Open Access under the University of Bristol's Scholarly Works Policy. The final published version (Version of Record) can be found on the publisher's website. The copyright of any third-party content, such as images, remains with the copyright holder.
Accepted author manuscript, 14.8 MBLicence: CC BY

Handle.net

Persistent link

Cite this

Price, W., & Damen, D. (2021). Play Fair: Frame Attributions in Video Models. In H. Ishikawa, C.-L. Liu, T. Pajdla, & J. Shi (Eds.), Computer Vision - ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part V (pp. 480-497). (Lecture Notes in Computer Science; Vol. 12626). Springer. https://doi.org/10.1007/978-3-030-69541-5_29

@inproceedings{775ca0b5f8af4fc997fe7b317a6d183d,

title = "Play Fair: Frame Attributions in Video Models",

abstract = "In this paper, we introduce an attribution method for explaining action recognition models. Such models fuse information from multiple frames within a video, through score aggregation or relational reasoning. We break down a model{\textquoteright}s class score into the sum of contributions from each frame, fairly. Our method adapts an axiomatic solution to fair reward distribution in cooperative games, known as the Shapley value, for elements in a variable-length sequence, which we call the Element Shapley Value (ESV). Critically, we propose a tractable approximation of ESV that scales linearly with the number of frames in the sequence. We employ ESV to explain two action recognition models (TRN and TSN) on the ﬁne-grained dataset Something-Something. We oﬀer detailed analysis of supporting/distracting frames, and the relationships of ESVs to the frame{\textquoteright}s position, class prediction, and sequence length. We compare ESV to naive baselines and two commonly used feature attribution methods: Grad-CAM and Integrated-Gradients.",

author = "Will Price and Dima Damen",

year = "2021",

month = feb,

day = "25",

doi = "10.1007/978-3-030-69541-5_29",

language = "English",

isbn = "9783030695408",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "480--497",

editor = "Hiroshi Ishikawa and Cheng-Lin Liu and Tomas Pajdla and Jianbo Shi",

booktitle = "Computer Vision - ACCV 2020",

note = "15th Asian Conference on Computer Vision ; Conference date: 30-11-2020 Through 04-12-2020",

url = "http://accv2020.kyoto/",

}

Price, W & Damen, D 2021, Play Fair: Frame Attributions in Video Models. in H Ishikawa, C-L Liu, T Pajdla & J Shi (eds), Computer Vision - ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part V. Lecture Notes in Computer Science, vol. 12626, Springer, pp. 480-497, 15th Asian Conference on Computer Vision, 30/11/20. https://doi.org/10.1007/978-3-030-69541-5_29

Play Fair: Frame Attributions in Video Models. / Price, Will; Damen, Dima .
Computer Vision - ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part V. ed. / Hiroshi Ishikawa; Cheng-Lin Liu; Tomas Pajdla; Jianbo Shi. Springer, 2021. p. 480-497 (Lecture Notes in Computer Science; Vol. 12626).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Play Fair

T2 - 15th Asian Conference on Computer Vision

AU - Price, Will

AU - Damen, Dima

PY - 2021/2/25

Y1 - 2021/2/25

N2 - In this paper, we introduce an attribution method for explaining action recognition models. Such models fuse information from multiple frames within a video, through score aggregation or relational reasoning. We break down a model’s class score into the sum of contributions from each frame, fairly. Our method adapts an axiomatic solution to fair reward distribution in cooperative games, known as the Shapley value, for elements in a variable-length sequence, which we call the Element Shapley Value (ESV). Critically, we propose a tractable approximation of ESV that scales linearly with the number of frames in the sequence. We employ ESV to explain two action recognition models (TRN and TSN) on the ﬁne-grained dataset Something-Something. We oﬀer detailed analysis of supporting/distracting frames, and the relationships of ESVs to the frame’s position, class prediction, and sequence length. We compare ESV to naive baselines and two commonly used feature attribution methods: Grad-CAM and Integrated-Gradients.

AB - In this paper, we introduce an attribution method for explaining action recognition models. Such models fuse information from multiple frames within a video, through score aggregation or relational reasoning. We break down a model’s class score into the sum of contributions from each frame, fairly. Our method adapts an axiomatic solution to fair reward distribution in cooperative games, known as the Shapley value, for elements in a variable-length sequence, which we call the Element Shapley Value (ESV). Critically, we propose a tractable approximation of ESV that scales linearly with the number of frames in the sequence. We employ ESV to explain two action recognition models (TRN and TSN) on the ﬁne-grained dataset Something-Something. We oﬀer detailed analysis of supporting/distracting frames, and the relationships of ESVs to the frame’s position, class prediction, and sequence length. We compare ESV to naive baselines and two commonly used feature attribution methods: Grad-CAM and Integrated-Gradients.

U2 - 10.1007/978-3-030-69541-5_29

DO - 10.1007/978-3-030-69541-5_29

M3 - Conference Contribution (Conference Proceeding)

SN - 9783030695408

T3 - Lecture Notes in Computer Science

SP - 480

EP - 497

BT - Computer Vision - ACCV 2020

A2 - Ishikawa, Hiroshi

A2 - Liu, Cheng-Lin

A2 - Pajdla, Tomas

A2 - Shi, Jianbo

PB - Springer

Y2 - 30 November 2020 through 4 December 2020

ER -

Play Fair: Frame Attributions in Video Models

Abstract

Publication series

Conference

Access to Document

Handle.net

Fingerprint

Cite this