EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Ahmad A K Dar Khalil; Dandan Shan; Bin Zhu; Jian Ma; Amlan Kar; Richard Higgins; David Fouhey; Sanja Fidler; Dima  Damen

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Ahmad A K Dar Khalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, David Fouhey, Sanja Fidler, Dima Damen

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

39 Downloads (Pure)

Abstract

We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 35 (NeurIPS 2022)
Editors	S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho
Publisher	Curran Associates, Inc
Pages	13745--13758
Number of pages	14
Volume	35
ISBN (Electronic)	9781713871088
Publication status	Published - 9 Dec 2022
Event	36th Conference on Neural Information Processing Systems - New Orleans, New Orleans, United States Duration: 28 Nov 2022 → 9 Dec 2022

Conference

Conference	36th Conference on Neural Information Processing Systems
Abbreviated title	NeurIPS 2022
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 9/12/22

Access to Document

Handle.net

Persistent link

8030 EPSRC via Oxford EP/T028572/1 Visual AI
Damen, D. (Principal Investigator)
1/12/20 → 30/11/25
Project: Research
UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS VISOR
Khalil, A. D. (Contributor), Shan, D. (Contributor), Zhu, B. (Contributor), Ma, J. (Contributor), Kar, A. (Contributor), Higgins, R. (Contributor), Fidler, S. (Contributor), Fouhey, D. (Contributor) & Aldamen, D. (Creator), University of Bristol, 11 Aug 2022
DOI: 10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7, http://data.bris.ac.uk/data/dataset/2v6cgv1x04ol22qp9rm9x2j6a7
Dataset
EPIC-KITCHENS-100
Aldamen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset
EPIC-KITCHENS 2018
Aldamen, D. (Creator), Moltisanti, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. (Creator), Fidler, S. (Contributor), Farinella, G. M. (Contributor) & Furnari, A. (Contributor), University of Bristol, 3 Apr 2018
DOI: 10.5523/bris.3h91syskeag572hl6tvuovwv4d, http://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d
Dataset

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

Dar Khalil, A. A. K., Shan, D., Zhu, B., Ma, J., Kar, A., Higgins, R., Fouhey, D., Fidler, S., & Damen, D. (2022). EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, & K. Cho (Eds.), Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (Vol. 35, pp. 13745--13758). Curran Associates, Inc. https://openreview.net/forum?id=djnKHOjpb7I

@inproceedings{c78d4405222649ce806f05561130a310,

title = "EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations",

abstract = "We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR",

author = "\{Dar Khalil\}, \{Ahmad A K\} and Dandan Shan and Bin Zhu and Jian Ma and Amlan Kar and Richard Higgins and David Fouhey and Sanja Fidler and Dima Damen",

year = "2022",

month = dec,

day = "9",

language = "English",

volume = "35",

pages = "13745----13758",

editor = "S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho",

booktitle = "Advances in Neural Information Processing Systems 35 (NeurIPS 2022)",

publisher = "Curran Associates, Inc",

note = "36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 09-12-2022",

}

Dar Khalil, AAK, Shan, D, Zhu, B, Ma, J, Kar, A, Higgins, R, Fouhey, D, Fidler, S & Damen, D 2022, EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. in S Koyejo, S Mohamed, A Agarwal, D Belgrave & K Cho (eds), Advances in Neural Information Processing Systems 35 (NeurIPS 2022). vol. 35, Curran Associates, Inc, pp. 13745--13758, 36th Conference on Neural Information Processing Systems, New Orleans, United States, 28/11/22. <https://openreview.net/forum?id=djnKHOjpb7I>

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. / Dar Khalil, Ahmad A K; Shan, Dandan; Zhu, Bin et al.
Advances in Neural Information Processing Systems 35 (NeurIPS 2022). ed. / S. Koyejo; S. Mohamed; A. Agarwal; D. Belgrave; K. Cho. Vol. 35 Curran Associates, Inc, 2022. p. 13745--13758.

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - EPIC-KITCHENS VISOR Benchmark

T2 - 36th Conference on Neural Information Processing Systems

AU - Dar Khalil, Ahmad A K

AU - Shan, Dandan

AU - Zhu, Bin

AU - Ma, Jian

AU - Kar, Amlan

AU - Higgins, Richard

AU - Fouhey, David

AU - Fidler, Sanja

AU - Damen, Dima

PY - 2022/12/9

Y1 - 2022/12/9

N2 - We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

AB - We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

M3 - Conference Contribution (Conference Proceeding)

VL - 35

SP - 13745

EP - 13758

BT - Advances in Neural Information Processing Systems 35 (NeurIPS 2022)

A2 - Koyejo, S.

A2 - Mohamed, S.

A2 - Agarwal, A.

A2 - Belgrave, D.

A2 - Cho, K.

PB - Curran Associates, Inc

Y2 - 28 November 2022 through 9 December 2022

ER -

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Abstract

Conference

Access to Document

Handle.net

Fingerprint

Projects

8030 EPSRC via Oxford EP/T028572/1 Visual AI

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Datasets

EPIC-KITCHENS VISOR

EPIC-KITCHENS-100

EPIC-KITCHENS 2018

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this