Epic-Sounds: A Large-Scale Dataset of Actions that Sound

Jaesung Huh; Jacob Chalk; Evangelos Kazakos; Dima  Damen; Andrew Zisserman

doi:10.1109/ICASSP49357.2023.10096198

Epic-Sounds: A Large-Scale Dataset of Actions that Sound

Jaesung Huh^*, Jacob Chalk^*, Evangelos Kazakos, Dima Damen, Andrew Zisserman

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

28 Downloads (Pure)

Abstract

We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos from EPIC-KITCHENS-100. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping free-form descriptions into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes, as well as 39.2k non-categorised segments, totalling 117.6k segments spanning 100 hours of audio, capturing diverse actions that sound in home kitchens. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.EPIC-SOUNDS and baseline source code is available from: https://epic-kitchens.github.io/epic-sounds.

Original language	English
Title of host publication	ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Number of pages	5
ISBN (Electronic)	9781728163277
ISBN (Print)	9781728163284
DOIs	https://doi.org/10.1109/ICASSP49357.2023.10096198
Publication status	Published - 5 May 2023
Event	2023 IEEE International Conference on Acoustics, Speech, and Signal Processing - Rodos Palace Luxury Convention Resort, Rhodes, Greece Duration: 4 Jun 2023 → 10 Jun 2023 https://2023.ieeeicassp.org/

Publication series

Name	Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing
ISSN (Print)	1520-6149
ISSN (Electronic)	2379-190X

Conference

Conference	2023 IEEE International Conference on Acoustics, Speech, and Signal Processing
Abbreviated title	ICASSP 2023
Country/Territory	Greece
City	Rhodes
Period	4/06/23 → 10/06/23
Internet address	https://2023.ieeeicassp.org/

Access to Document

10.1109/ICASSP49357.2023.10096198Licence: Unspecified

Full-text PDF (accepted manuscript)
This is the accepted author manuscript (AAM). Copyright for the images within is retained by the author. The final published version (version of record) is available online via IEEE at https://doi.org/10.1109/ICASSP49357.2023.10096198.  Please refer to any applicable terms of use of the publisher.
Accepted author manuscript, 5.47 MB

https://arxiv.org/abs/2302.00646v1Licence: CC BY

Handle.net

Persistent link

1 Finished
1 Active

Visual AI - Full Programme Grant Extension
Damen, D. (Principal Investigator)
1/06/23 → 30/11/25
Project: Research
UMPIRE: United Model for the Perception of Interactions for visual Recognition
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS-100
Aldamen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities
Alam, S. R. (Manager), Williams, D. A. G. (Manager), Eccleston, P. E. (Manager) & Greene, D. (Manager)
Facility/equipment: Facility

Cite this

Huh, J., Chalk, J., Kazakos, E., Damen, D., & Zisserman, A. (2023). Epic-Sounds: A Large-Scale Dataset of Actions that Sound. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ICASSP49357.2023.10096198

Huh, Jaesung ; Chalk, Jacob ; Kazakos, Evangelos et al. / Epic-Sounds : A Large-Scale Dataset of Actions that Sound. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers (IEEE), 2023. (Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing).

@inproceedings{41636e4683f04ecc8e308179679d463e,

title = "Epic-Sounds: A Large-Scale Dataset of Actions that Sound",

abstract = "We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos from EPIC-KITCHENS-100. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping free-form descriptions into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes, as well as 39.2k non-categorised segments, totalling 117.6k segments spanning 100 hours of audio, capturing diverse actions that sound in home kitchens. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.EPIC-SOUNDS and baseline source code is available from: https://epic-kitchens.github.io/epic-sounds.",

author = "Jaesung Huh and Jacob Chalk and Evangelos Kazakos and Dima Damen and Andrew Zisserman",

year = "2023",

month = may,

day = "5",

doi = "10.1109/ICASSP49357.2023.10096198",

language = "English",

isbn = "9781728163284",

series = "Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

booktitle = "ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

address = "United States",

note = "2023 IEEE International Conference on Acoustics, Speech, and Signal Processing , ICASSP 2023 ; Conference date: 04-06-2023 Through 10-06-2023",

url = "https://2023.ieeeicassp.org/",

}

Huh, J, Chalk, J, Kazakos, E, Damen, D & Zisserman, A 2023, Epic-Sounds: A Large-Scale Dataset of Actions that Sound. in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing, Institute of Electrical and Electronics Engineers (IEEE), 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing , Rhodes, Greece, 4/06/23. https://doi.org/10.1109/ICASSP49357.2023.10096198

Epic-Sounds: A Large-Scale Dataset of Actions that Sound. / Huh, Jaesung; Chalk, Jacob; Kazakos, Evangelos et al.
ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers (IEEE), 2023. (Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Epic-Sounds

T2 - 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing

AU - Huh, Jaesung

AU - Chalk, Jacob

AU - Kazakos, Evangelos

AU - Damen, Dima

AU - Zisserman, Andrew

PY - 2023/5/5

Y1 - 2023/5/5

N2 - We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos from EPIC-KITCHENS-100. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping free-form descriptions into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes, as well as 39.2k non-categorised segments, totalling 117.6k segments spanning 100 hours of audio, capturing diverse actions that sound in home kitchens. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.EPIC-SOUNDS and baseline source code is available from: https://epic-kitchens.github.io/epic-sounds.

AB - We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos from EPIC-KITCHENS-100. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping free-form descriptions into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes, as well as 39.2k non-categorised segments, totalling 117.6k segments spanning 100 hours of audio, capturing diverse actions that sound in home kitchens. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.EPIC-SOUNDS and baseline source code is available from: https://epic-kitchens.github.io/epic-sounds.

U2 - 10.1109/ICASSP49357.2023.10096198

DO - 10.1109/ICASSP49357.2023.10096198

M3 - Conference Contribution (Conference Proceeding)

SN - 9781728163284

T3 - Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing

BT - ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 4 June 2023 through 10 June 2023

ER -

Huh J, Chalk J, Kazakos E, Damen D, Zisserman A. Epic-Sounds: A Large-Scale Dataset of Actions that Sound. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers (IEEE). 2023. (Proceedings of the ... IEEE International Conference on Acoustics, Speech, and Signal Processing). doi: 10.1109/ICASSP49357.2023.10096198

Epic-Sounds: A Large-Scale Dataset of Actions that Sound

Abstract

Publication series

Conference

Access to Document

Handle.net

Fingerprint

Projects

Visual AI - Full Programme Grant Extension

UMPIRE: United Model for the Perception of Interactions for visual Recognition

Datasets

EPIC-KITCHENS-100

Equipment

HPC (High Performance Computing) and HTC (High Throughput Computing) Facilities

Cite this