EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

Ahmad A K Dar Khalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, David Fouhey, Sanja Fidler, Dima Damen

Research output: Contribution to conferenceConference Paperpeer-review

13 Downloads (Pure)


We introduce VISOR, a new dataset of pixel annotations and a benchmark suite
for segmenting hands and active objects in egocentric video. VISOR annotates
videos from EPIC-KITCHENS, which comes with a new set of challenges not
encountered in current video segmentation datasets. Specifically, we need to ensure
both short- and long-term consistency of pixel-level annotations as objects undergo
transformative interactions, e.g. an onion is peeled, diced and cooked - where we
aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping
board, knife, pan, as well as the acting hands. VISOR introduces an annotation
pipeline, AI-powered in parts, for scalability and quality. In total, we publicly
release 272K manual semantic masks of 257 object classes, 9.9M interpolated
dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed
videos. Along with the annotations, we introduce three challenges in video object
segmentation, interaction understanding and long-term reasoning.
For data, code and leaderboards: http://epic-kitchens.github.io/VISOR
Original languageEnglish
Number of pages52
Publication statusPublished - 9 Dec 2022
EventConference on Neural Information Processing Systems - New Orleans, New Orleans, United States
Duration: 28 Nov 20229 Dec 2022


ConferenceConference on Neural Information Processing Systems
Abbreviated titleNeurIPS
Country/TerritoryUnited States
CityNew Orleans


Dive into the research topics of 'EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations'. Together they form a unique fingerprint.
  • HPC (High Performance Computing) Facility

    Sadaf R Alam (Manager), Steven A Chapman (Manager), Polly E Eccleston (Other), Simon H Atack (Other) & D A G Williams (Manager)

    Facility/equipment: Facility

Cite this