UnweaveNet: Unweaving Activity Stories

Will Price; Carl Vondrick; Dima  Damen

doi:10.48550/arXiv.2112.10194

UnweaveNet: Unweaving Activity Stories

Will Price, Carl Vondrick, Dima Damen^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

8 Downloads (Pure)

Abstract

Our lives can be seen as a complex weaving of activi- ties; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us. Observing a video of unscripted daily activities, we parse the video into its constituent activity threads through a process we call unweaving. To accomplish this, we introduce a video representation explicitly capturing activity threads called a thread bank, along with a neural controller capable of detecting goal changes and resuming of past activities, together forming UnweaveNet. We train and evaluate UnweaveNet on sequences from the unscripted egocentric dataset EPIC-KITCHENS. We propose and showcase the effi- cacy of pretraining UnweaveNet in a self-supervised manner.

Original language	English
Title of host publication	2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	13770-13779
Number of pages	10
ISBN (Electronic)	978-1-6654-6946-3
ISBN (Print)	978-1-6654-6947-0
DOIs	https://doi.org/10.48550/arXiv.2112.10194 https://doi.org/10.1109/CVPR52688.2022.01340
Publication status	Unpublished - 24 Jun 2022
Event	Computer Vision and Pattern Recognition (CVPR) - New Orleans, United States Duration: 19 Jun 2022 → 25 Jun 2022

Publication series

Name	Proceedings. IEEE Conference on Computer Vision and Pattern Recognition
Publisher	IEEE
ISSN (Print)	1063-6919
ISSN (Electronic)	2575-7075

Conference

Conference	Computer Vision and Pattern Recognition (CVPR)
Country/Territory	United States
Period	19/06/22 → 25/06/22

Access to Document

Full-text PDF (accepted author manuscript)
This is the accepted author manuscript (AAM).
Accepted author manuscript, 19.4 MB

https://openaccess.thecvf.com/content/CVPR2022/html/Price_UnweaveNet_Unweaving_Activity_Stories_CVPR_2022_paper.html

Handle.net

Persistent link

UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D.
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS-100
Damen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. J. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset

Cite this

Price, W., Vondrick, C., & Damen, D. (2022). UnweaveNet: Unweaving Activity Stories. Unpublished. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 13770-13779). (Proceedings. IEEE Conference on Computer Vision and Pattern Recognition). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.48550/arXiv.2112.10194, https://doi.org/10.1109/CVPR52688.2022.01340

@inproceedings{8aacf3a81873471da223f2770476ead8,

title = "UnweaveNet: Unweaving Activity Stories",

abstract = "Our lives can be seen as a complex weaving of activi- ties; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us. Observing a video of unscripted daily activities, we parse the video into its constituent activity threads through a process we call unweaving. To accomplish this, we introduce a video representation explicitly capturing activity threads called a thread bank, along with a neural controller capable of detecting goal changes and resuming of past activities, together forming UnweaveNet. We train and evaluate UnweaveNet on sequences from the unscripted egocentric dataset EPIC-KITCHENS. We propose and showcase the effi- cacy of pretraining UnweaveNet in a self-supervised manner.",

author = "Will Price and Carl Vondrick and Dima Damen",

year = "2022",

month = jun,

day = "24",

doi = "10.48550/arXiv.2112.10194",

language = "English",

isbn = "978-1-6654-6947-0",

series = "Proceedings. IEEE Conference on Computer Vision and Pattern Recognition",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "13770--13779",

booktitle = "2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)",

address = "United States",

note = "Computer Vision and Pattern Recognition (CVPR) ; Conference date: 19-06-2022 Through 25-06-2022",

}

Price, W, Vondrick, C & Damen, D 2022, UnweaveNet: Unweaving Activity Stories. in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Proceedings. IEEE Conference on Computer Vision and Pattern Recognition, Institute of Electrical and Electronics Engineers (IEEE), pp. 13770-13779, Computer Vision and Pattern Recognition (CVPR), United States, 19/06/22. https://doi.org/10.48550/arXiv.2112.10194, https://doi.org/10.1109/CVPR52688.2022.01340

UnweaveNet: Unweaving Activity Stories. / Price, Will; Vondrick, Carl; Damen, Dima .
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Institute of Electrical and Electronics Engineers (IEEE), 2022. p. 13770-13779 (Proceedings. IEEE Conference on Computer Vision and Pattern Recognition).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - UnweaveNet

T2 - Computer Vision and Pattern Recognition (CVPR)

AU - Price, Will

AU - Vondrick, Carl

AU - Damen, Dima

PY - 2022/6/24

Y1 - 2022/6/24

N2 - Our lives can be seen as a complex weaving of activi- ties; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us. Observing a video of unscripted daily activities, we parse the video into its constituent activity threads through a process we call unweaving. To accomplish this, we introduce a video representation explicitly capturing activity threads called a thread bank, along with a neural controller capable of detecting goal changes and resuming of past activities, together forming UnweaveNet. We train and evaluate UnweaveNet on sequences from the unscripted egocentric dataset EPIC-KITCHENS. We propose and showcase the effi- cacy of pretraining UnweaveNet in a self-supervised manner.

AB - Our lives can be seen as a complex weaving of activi- ties; we switch from one activity to another, to maximise our achievements or in reaction to demands placed upon us. Observing a video of unscripted daily activities, we parse the video into its constituent activity threads through a process we call unweaving. To accomplish this, we introduce a video representation explicitly capturing activity threads called a thread bank, along with a neural controller capable of detecting goal changes and resuming of past activities, together forming UnweaveNet. We train and evaluate UnweaveNet on sequences from the unscripted egocentric dataset EPIC-KITCHENS. We propose and showcase the effi- cacy of pretraining UnweaveNet in a self-supervised manner.

U2 - 10.48550/arXiv.2112.10194

DO - 10.48550/arXiv.2112.10194

M3 - Conference Contribution (Conference Proceeding)

SN - 978-1-6654-6947-0

T3 - Proceedings. IEEE Conference on Computer Vision and Pattern Recognition

SP - 13770

EP - 13779

BT - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 19 June 2022 through 25 June 2022

ER -

UnweaveNet: Unweaving Activity Stories

Abstract

Publication series

Conference

Access to Document

Handle.net

Fingerprint

Projects

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Datasets

EPIC-KITCHENS-100

Cite this