EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Ahmad A K Dar Khalil; Dandan Shan; Bin Zhu; Jian Ma; Amlan Kar; Richard Higgins; David Fouhey; Sanja Fidler; Dima  Damen

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Ahmad A K Dar Khalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, David Fouhey, Sanja Fidler, Dima Damen

Research output: Contribution to conference › Conference Paper › peer-review

13 Downloads (Pure)

Abstract

We introduce VISOR, a new dataset of pixel annotations and a benchmark suite
for segmenting hands and active objects in egocentric video. VISOR annotates
videos from EPIC-KITCHENS, which comes with a new set of challenges not
encountered in current video segmentation datasets. Specifically, we need to ensure
both short- and long-term consistency of pixel-level annotations as objects undergo
transformative interactions, e.g. an onion is peeled, diced and cooked - where we
aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping
board, knife, pan, as well as the acting hands. VISOR introduces an annotation
pipeline, AI-powered in parts, for scalability and quality. In total, we publicly
release 272K manual semantic masks of 257 object classes, 9.9M interpolated
dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed
videos. Along with the annotations, we introduce three challenges in video object
segmentation, interaction understanding and long-term reasoning.
For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

Original language	English
Number of pages	52
Publication status	Published - 9 Dec 2022
Event	Conference on Neural Information Processing Systems - New Orleans, New Orleans, United States Duration: 28 Nov 2022 → 9 Dec 2022

Conference

Conference	Conference on Neural Information Processing Systems
Abbreviated title	NeurIPS
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 9/12/22

Access to Document

Full-text PDF (final published version)Final published version, 74.2 MB

Handle.net

Persistent link

8030 EPSRC via Oxford EP/T028572/1 Visual AI
Damen, D.
1/12/20 → 30/11/25
Project: Research
UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D.
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS VISOR
Khalil, A. D. (Contributor), Shan, D. (Contributor), Zhu, B. (Contributor), Ma, J. (Contributor), Kar, A. (Contributor), Higgins, R. (Contributor), Fidler, S. (Contributor), Fouhey, D. (Contributor) & Damen, D. (Creator), University of Bristol, 11 Aug 2022
DOI: 10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7, http://data.bris.ac.uk/data/dataset/2v6cgv1x04ol22qp9rm9x2j6a7
Dataset
EPIC-KITCHENS-100
Damen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. J. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset
EPIC-KITCHENS 2018
Damen, D. (Creator), Moltisanti, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. J. (Creator), Fidler, S. (Contributor), Farinella, G. M. (Contributor) & Furnari, A. (Contributor), University of Bristol, 3 Apr 2018
DOI: 10.5523/bris.3h91syskeag572hl6tvuovwv4d, http://data.bris.ac.uk/data/dataset/3h91syskeag572hl6tvuovwv4d
Dataset

HPC (High Performance Computing) Facility
Sadaf R Alam (Manager), Steven A Chapman (Manager), Polly E Eccleston (Other), Simon H Atack (Other) & D A G Williams (Manager)
Facility/equipment: Facility

Cite this

@conference{c78d4405222649ce806f05561130a310,

title = "EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations",

abstract = "We introduce VISOR, a new dataset of pixel annotations and a benchmark suitefor segmenting hands and active objects in egocentric video. VISOR annotatesvideos from EPIC-KITCHENS, which comes with a new set of challenges notencountered in current video segmentation datasets. Specifically, we need to ensureboth short- and long-term consistency of pixel-level annotations as objects undergotransformative interactions, e.g. an onion is peeled, diced and cooked - where weaim to obtain accurate pixel-level annotations of the peel, onion pieces, choppingboard, knife, pan, as well as the acting hands. VISOR introduces an annotationpipeline, AI-powered in parts, for scalability and quality. In total, we publiclyrelease 272K manual semantic masks of 257 object classes, 9.9M interpolateddense masks, 67K hand-object relations, covering 36 hours of 179 untrimmedvideos. Along with the annotations, we introduce three challenges in video objectsegmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR",

author = "{Dar Khalil}, {Ahmad A K} and Dandan Shan and Bin Zhu and Jian Ma and Amlan Kar and Richard Higgins and David Fouhey and Sanja Fidler and Dima Damen",

year = "2022",

month = dec,

day = "9",

language = "English",

note = "Conference on Neural Information Processing Systems, NeurIPS ; Conference date: 28-11-2022 Through 09-12-2022",

}

TY - CONF

T1 - EPIC-KITCHENS VISOR Benchmark

T2 - Conference on Neural Information Processing Systems

AU - Dar Khalil, Ahmad A K

AU - Shan, Dandan

AU - Zhu, Bin

AU - Ma, Jian

AU - Kar, Amlan

AU - Higgins, Richard

AU - Fouhey, David

AU - Fidler, Sanja

AU - Damen, Dima

PY - 2022/12/9

Y1 - 2022/12/9

N2 - We introduce VISOR, a new dataset of pixel annotations and a benchmark suitefor segmenting hands and active objects in egocentric video. VISOR annotatesvideos from EPIC-KITCHENS, which comes with a new set of challenges notencountered in current video segmentation datasets. Specifically, we need to ensureboth short- and long-term consistency of pixel-level annotations as objects undergotransformative interactions, e.g. an onion is peeled, diced and cooked - where weaim to obtain accurate pixel-level annotations of the peel, onion pieces, choppingboard, knife, pan, as well as the acting hands. VISOR introduces an annotationpipeline, AI-powered in parts, for scalability and quality. In total, we publiclyrelease 272K manual semantic masks of 257 object classes, 9.9M interpolateddense masks, 67K hand-object relations, covering 36 hours of 179 untrimmedvideos. Along with the annotations, we introduce three challenges in video objectsegmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

AB - We introduce VISOR, a new dataset of pixel annotations and a benchmark suitefor segmenting hands and active objects in egocentric video. VISOR annotatesvideos from EPIC-KITCHENS, which comes with a new set of challenges notencountered in current video segmentation datasets. Specifically, we need to ensureboth short- and long-term consistency of pixel-level annotations as objects undergotransformative interactions, e.g. an onion is peeled, diced and cooked - where weaim to obtain accurate pixel-level annotations of the peel, onion pieces, choppingboard, knife, pan, as well as the acting hands. VISOR introduces an annotationpipeline, AI-powered in parts, for scalability and quality. In total, we publiclyrelease 272K manual semantic masks of 257 object classes, 9.9M interpolateddense masks, 67K hand-object relations, covering 36 hours of 179 untrimmedvideos. Along with the annotations, we introduce three challenges in video objectsegmentation, interaction understanding and long-term reasoning.For data, code and leaderboards: http://epic-kitchens.github.io/VISOR

M3 - Conference Paper

Y2 - 28 November 2022 through 9 December 2022

ER -

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Abstract

Conference

Access to Document

Handle.net

Fingerprint

Projects

8030 EPSRC via Oxford EP/T028572/1 Visual AI

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Datasets

EPIC-KITCHENS VISOR

EPIC-KITCHENS-100

EPIC-KITCHENS 2018

Equipment

HPC (High Performance Computing) Facility

Cite this