Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

Dima  Damen; Hazel R Doughty; Giovanni Maria Farinella; Antonino Furnari; Vangelis Kazakos; Jian Ma; Davide Moltisanti; Jonathan P N Munro; Toby J Perrett; Will Price; Michael Wray

doi:10.1007/s11263-021-01531-2

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

Dima Damen^*, Hazel R Doughty, Giovanni Maria Farinella, Antonino Furnari, Vangelis Kazakos, Jian Ma, Davide Moltisanti, Jonathan P N Munro, Toby J Perrett, Will Price, Michael Wray

^*Corresponding author for this work

Research output: Contribution to journal › Article (Academic Journal) › peer-review

95 Citations (Scopus)

146 Downloads (Pure)

Abstract

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras.
Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments).
This collection enables new challenges such as action detection and evaluating the "test of time'' - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later.

The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics

Original language	English
Pages (from-to)	33–55
Number of pages	23
Journal	International Journal of Computer Vision (IJCV)
Volume	130
Issue number	1
Early online date	20 Oct 2021
DOIs	https://doi.org/10.1007/s11263-021-01531-2
Publication status	Published - Jan 2022

Bibliographical note

Funding Information:
Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset.

Funding Information:
Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset.

Publisher Copyright:
© 2021, The Author(s).

Keywords

Video Dataset
Egocentric Vision
Action Understanding
computer vision

Access to Document

10.1007/s11263-021-01531-2

Full-text PDF (final published version)
This is the final published version of the article (version of record). It first appeared online via Springer at 10.1007/s11263-021-01531-2. Please refer to any applicable terms of use of the publisher.
Final published version, 4.78 MBLicence: CC BY

https://arxiv.org/abs/2006.13256

Handle.net

Persistent link

Embargoed Document

Full-text PDF (accepted author manuscript)
Accepted author manuscript, 26.5 MB
Request copy

95 Citations
1 Conference Contribution (Conference Proceeding)
1 Article (Academic Journal)

The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Damen, D., Doughty, H. R., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J. P. N., Perrett, T., Price, W. & Wray, M., 6 May 2020, (E-pub ahead of print) In: IEEE Transactions on Pattern Analysis and Machine Intelligence. 0, p. 1-16 16 p.
Research output: Contribution to journal › Article (Academic Journal) › peer-review

Open Access
62 Citations (Scopus)
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G., Fidler, S., Furnari, A., Kazakos, V., Moltisanti, D., Munro, J., Perrett, T. & Wray, M., 6 Oct 2018, Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV. Springer, Cham, p. 753-771 19 p. (Lecture Notes in Computer Science; vol. 11208).
Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Open Access
File
382 Citations (Scopus)

339 Downloads (Pure)

1 Active

UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D.
1/02/20 → 31/01/25
Project: Research

Temporal Labelling for Action Recognition in Videos
Author: Moltisanti, D., 28 Nov 2019
Supervisor: Aldamen, D. (Supervisor) & Mayol-Cuevas, W. (Supervisor)
Student thesis: Doctoral Thesis › Doctor of Philosophy (PhD)

File
Verbs and Me: An Investigation Into Verbs as Labels for Action Recognition in Video Understanding
Author: Wray, M., 23 Jan 2020
Supervisor: Damen, D. (Supervisor)
Student thesis: Doctoral Thesis › Doctor of Philosophy (PhD)

File

EPIC-KITCHENS-100
Damen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. J. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset

HPC (High Performance Computing) Facility
Sadaf R Alam (Manager), Steven A Chapman (Manager), Polly E Eccleston (Other), Simon H Atack (Other) & D A G Williams (Manager)
Facility/equipment: Facility

Cite this

@article{d7ae2581721d406ebd05449f20132386,

title = "Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100",

abstract = "This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments).This collection enables new challenges such as action detection and evaluating the {"}test of time'' - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later.The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics",

keywords = "Video Dataset, Egocentric Vision, Action Understanding, computer vision",

author = "Dima Damen and Doughty, {Hazel R} and Farinella, {Giovanni Maria} and Antonino Furnari and Vangelis Kazakos and Jian Ma and Davide Moltisanti and Munro, {Jonathan P N} and Perrett, {Toby J} and Will Price and Michael Wray",

note = "Funding Information: Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset. Funding Information: Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset. Publisher Copyright: {\textcopyright} 2021, The Author(s).",

year = "2022",

month = jan,

doi = "10.1007/s11263-021-01531-2",

language = "English",

volume = "130",

pages = "33–55",

journal = "International Journal of Computer Vision (IJCV)",

number = "1",

}

TY - JOUR

T1 - Rescaling Egocentric Vision

T2 - Collection Pipeline and Challenges for EPIC-KITCHENS-100

AU - Damen, Dima

AU - Doughty, Hazel R

AU - Farinella, Giovanni Maria

AU - Furnari, Antonino

AU - Kazakos, Vangelis

AU - Ma, Jian

AU - Moltisanti, Davide

AU - Munro, Jonathan P N

AU - Perrett, Toby J

AU - Price, Will

AU - Wray, Michael

N1 - Funding Information: Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset. Funding Information: Research at Bristol is supported by Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE (EP/T004991/1). Research at Catania is sponsored by Piano della Ricerca 2016-2018 linea di Intervento 2 of DMI, by MISE - PON I&C 2014-2020, ENIGMA project (CUP: B61B19000520008) and by MIUR AIM - Attrazione e Mobilita Internazionale Linea 1 - AIM1893589 - CUP E64118002540007. We thank David Fouhey and Dandan Shan from University of Michigan for providing the ego-trained hand-object detection model prior to its public release. We also thank Sanja Fidler from University of Toronto for contributing to the first edition of EPIC-KITCHENS. We appreciate the efforts of all voluntary participants to collect and narrate this dataset. Publisher Copyright: © 2021, The Author(s).

PY - 2022/1

Y1 - 2022/1

N2 - This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments).This collection enables new challenges such as action detection and evaluating the "test of time'' - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later.The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics

AB - This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version, EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments).This collection enables new challenges such as action detection and evaluating the "test of time'' - i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later.The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics

KW - Video Dataset

KW - Egocentric Vision

KW - Action Understanding

KW - computer vision

U2 - 10.1007/s11263-021-01531-2

DO - 10.1007/s11263-021-01531-2

M3 - Article (Academic Journal)

VL - 130

SP - 33

EP - 55

JO - International Journal of Computer Vision (IJCV)

JF - International Journal of Computer Vision (IJCV)

IS - 1

ER -

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

Abstract

Bibliographical note

Keywords

Access to Document

Handle.net

Embargoed Document

Fingerprint

Research output

The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Projects

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Student theses

Temporal Labelling for Action Recognition in Videos

Verbs and Me: An Investigation Into Verbs as Labels for Action Recognition in Video Understanding

Datasets

EPIC-KITCHENS-100

Equipment

HPC (High Performance Computing) Facility

Cite this