Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Dima Damen; Hazel Doughty; Giovanni Farinella; Sanja Fidler; Antonio Furnari; Vangelis Kazakos; Davide Moltisanti; Jonathan Munro; Toby Perrett; Michael Wray

doi:10.1007/978-3-030-01225-0_44

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Dima Damen, Hazel Doughty, Giovanni Farinella, Sanja Fidler, Antonio Furnari, Vangelis Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Michael Wray

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

424 Citations (Scopus)

577 Downloads (Pure)

Abstract

First-person vision is gaining interest as it offers a unique viewpoint on people’s interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55h of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

Original language	English
Title of host publication	Computer Vision – ECCV 2018
Subtitle of host publication	15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV
Publisher	Springer, Cham
Pages	753-771
Number of pages	19
ISBN (Electronic)	9783030012250
ISBN (Print)	9783030012243
DOIs	https://doi.org/10.1007/978-3-030-01225-0_44
Publication status	Published - 6 Oct 2018
Event	European Conference on Computer Vision - Munich, Germany Duration: 7 Sept 2018 → 14 Sept 2018

Publication series

Name	Lecture Notes in Computer Science
Volume	11208
ISSN (Print)	0302-9743

Conference

Conference	European Conference on Computer Vision
Abbreviated title	ECCV
Country/Territory	Germany
City	Munich
Period	7/09/18 → 14/09/18

Research Groups and Themes

Jean Golding

Keywords

Egocentric vision
Dataset
Benchmarks
First-person vision
Egocentric object detection
Action recognition and anticipation

Access to Document

10.1007/978-3-030-01225-0_44

Full-text PDF (accepted author manuscript)
This is the author accepted manuscript (AAM). The final published version (version of record) is available online via Springer at https://link.springer.com/chapter/10.1007/978-3-030-01225-0_44 . Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 2.88 MB

Handle.net

Persistent link

Cite this

Damen, D., Doughty, H., Farinella, G., Fidler, S., Furnari, A., Kazakos, V., Moltisanti, D., Munro, J., Perrett, T., & Wray, M. (2018). Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV (pp. 753-771). (Lecture Notes in Computer Science; Vol. 11208). Springer, Cham. https://doi.org/10.1007/978-3-030-01225-0_44

@inproceedings{b25d2e1d9af24f909f6c668dcd6fc450,

title = "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset",

abstract = "First-person vision is gaining interest as it offers a unique viewpoint on people{\textquoteright}s interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55h of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.",

keywords = "Egocentric vision, Dataset, Benchmarks, First-person vision, Egocentric object detection, Action recognition and anticipation",

author = "Dima Damen and Hazel Doughty and Giovanni Farinella and Sanja Fidler and Antonio Furnari and Vangelis Kazakos and Davide Moltisanti and Jonathan Munro and Toby Perrett and Michael Wray",

year = "2018",

month = oct,

day = "6",

doi = "10.1007/978-3-030-01225-0\_44",

language = "English",

isbn = "9783030012243",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Cham",

pages = "753--771",

booktitle = "Computer Vision – ECCV 2018",

address = "Switzerland",

note = "European Conference on Computer Vision, ECCV ; Conference date: 07-09-2018 Through 14-09-2018",

}

Damen, D, Doughty, H, Farinella, G, Fidler, S, Furnari, A, Kazakos, V, Moltisanti, D, Munro, J, Perrett, T & Wray, M 2018, Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. in Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV. Lecture Notes in Computer Science, vol. 11208, Springer, Cham, pp. 753-771, European Conference on Computer Vision, Munich, Germany, 7/09/18. https://doi.org/10.1007/978-3-030-01225-0_44

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. / Damen, Dima; Doughty, Hazel; Farinella, Giovanni et al.
Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV. Springer, Cham, 2018. p. 753-771 (Lecture Notes in Computer Science; Vol. 11208).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Scaling Egocentric Vision

T2 - European Conference on Computer Vision

AU - Damen, Dima

AU - Doughty, Hazel

AU - Farinella, Giovanni

AU - Fidler, Sanja

AU - Furnari, Antonio

AU - Kazakos, Vangelis

AU - Moltisanti, Davide

AU - Munro, Jonathan

AU - Perrett, Toby

AU - Wray, Michael

PY - 2018/10/6

Y1 - 2018/10/6

N2 - First-person vision is gaining interest as it offers a unique viewpoint on people’s interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55h of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

AB - First-person vision is gaining interest as it offers a unique viewpoint on people’s interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55h of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

KW - Egocentric vision

KW - Dataset

KW - Benchmarks

KW - First-person vision

KW - Egocentric object detection

KW - Action recognition and anticipation

UR - https://arxiv.org/abs/1804.02748

U2 - 10.1007/978-3-030-01225-0_44

DO - 10.1007/978-3-030-01225-0_44

M3 - Conference Contribution (Conference Proceeding)

SN - 9783030012243

T3 - Lecture Notes in Computer Science

SP - 753

EP - 771

BT - Computer Vision – ECCV 2018

PB - Springer, Cham

Y2 - 7 September 2018 through 14 September 2018

ER -

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Abstract

Publication series

Conference

Research Groups and Themes

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

LOCATE: LOcation adaptive Constrained Activity recognition using Transfer learning

GLANCE - EPSRC Call for Outlines - User Interaction with ICT

GLANCE: GLANCE: GLAnceable Nuances for Contextual Events

Verbs and Me: An Investigation Into Verbs as Labels for Action Recognition in Video Understanding

Professor Dima Damen

Cite this

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Abstract

Publication series

Conference

Research Groups and Themes

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Research output

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

Projects

LOCATE: LOcation adaptive Constrained Activity recognition using Transfer learning

GLANCE - EPSRC Call for Outlines - User Interaction with ICT

GLANCE: GLANCE: GLAnceable Nuances for Contextual Events

Student theses

Verbs and Me: An Investigation Into Verbs as Labels for Action Recognition in Video Understanding

Profiles

Professor Dima Damen

Cite this