AMEGO: Active Memory from long EGOcentric videos

Gabriele Goletto; Tushar Nagarajan; Giuseppe Averta; Dima  Damen

doi:10.1007/978-3-031-72624-8_6

AMEGO: Active Memory from long EGOcentric videos

Gabriele Goletto^*, Tushar Nagarajan, Giuseppe Averta, Dima Damen

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

14 Downloads (Pure)

Abstract

Egocentric videos provide a unique perspective into individuals’ daily experiences, yet their unstructured nature presents challenges for perception. In this paper, we introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human’s ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate our understanding of very-long egocentric videos, we introduce the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. We showcase improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin.

Original language	English
Title of host publication	Computer Vision – ECCV 2024
Publisher	Springer, Cham
Pages	92-110
ISBN (Electronic)	9783031726248
ISBN (Print)	9783031726231
DOIs	https://doi.org/10.1007/978-3-031-72624-8_6
Publication status	E-pub ahead of print - 26 Oct 2024
Event	The 18th European Conference on Computer Vision ECCV 2024 - MiCo Milano, Milano, Italy Duration: 29 Sept 2024 → 4 Oct 2024 https://eccv2024.ecva.net/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	15071
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The 18th European Conference on Computer Vision ECCV 2024
Abbreviated title	ECCV 2024
Country/Territory	Italy
City	Milano
Period	29/09/24 → 4/10/24
Internet address	https://eccv2024.ecva.net/

Bibliographical note

Publisher Copyright:
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG.

Access to Document

10.1007/978-3-031-72624-8_6

Full-text PDF (accepted manuscript)
This is the accepted author manuscript (AAM) of the article which has been made Open Access under the University of Bristol's Scholarly Works Policy. The final published version (Version of Record) can be found on the publisher's website. The copyright of any third-party content, such as images, remains with the copyright holder.
Accepted author manuscript, 6.23 MBLicence: CC BY

Handle.net

Persistent link

1 Article (Academic Journal)

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100
Damen, D., Doughty, H. R., Farinella, G. M., Furnari, A., Kazakos, V., Ma, J., Moltisanti, D., Munro, J. P. N., Perrett, T. J., Price, W. & Wray, M., Jan 2022, In: International Journal of Computer Vision (IJCV). 130, 1, p. 33–55 23 p.
Research output: Contribution to journal › Article (Academic Journal) › peer-review

Open Access
File
330 Citations (Scopus)

395 Downloads (Pure)

1 Finished
1 Active

8030 EPSRC via Oxford EP/T028572/1 Visual AI
Damen, D. (Principal Investigator)
1/12/20 → 30/11/25
Project: Research, Parent
8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship
Damen, D. (Principal Investigator)
1/02/20 → 31/01/25
Project: Research

EPIC-KITCHENS VISOR
Khalil, A. D. (Contributor), Shan, D. (Contributor), Zhu, B. (Contributor), Ma, J. (Contributor), Kar, A. (Contributor), Higgins, R. (Contributor), Fidler, S. (Contributor), Fouhey, D. (Contributor) & Aldamen, D. (Creator), University of Bristol, 11 Aug 2022
DOI: 10.5523/bris.2v6cgv1x04ol22qp9rm9x2j6a7, http://data.bris.ac.uk/data/dataset/2v6cgv1x04ol22qp9rm9x2j6a7
Dataset
EPIC-KITCHENS-100
Aldamen, D. (Creator), Kazakos, E. (Creator), Doughty, H. (Creator), Munro, J. (Creator), Price, W. (Creator), Wray, M. (Creator), Perrett, T. (Creator) & Ma, J. (Creator), University of Bristol, 15 May 2020
DOI: 10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m, http://data.bris.ac.uk/data/dataset/2g1n6qdydwa9u22shpxqzp0t8m
Dataset

Cite this

@inproceedings{fa1db34c80a1493886c8925180941dbf,

title = "AMEGO: Active Memory from long EGOcentric videos",

abstract = "Egocentric videos provide a unique perspective into individuals{\textquoteright} daily experiences, yet their unstructured nature presents challenges for perception. In this paper, we introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human{\textquoteright}s ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate our understanding of very-long egocentric videos, we introduce the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. We showcase improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin.",

author = "Gabriele Goletto and Tushar Nagarajan and Giuseppe Averta and Dima Damen",

note = "Publisher Copyright: {\textcopyright} 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG.; The 18th European Conference on Computer Vision ECCV 2024, ECCV 2024 ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2024",

month = oct,

day = "26",

doi = "10.1007/978-3-031-72624-8_6",

language = "English",

isbn = "9783031726231",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Cham",

pages = "92--110",

booktitle = "Computer Vision – ECCV 2024",

address = "Switzerland",

url = "https://eccv2024.ecva.net/",

}

TY - GEN

T1 - AMEGO

T2 - The 18th European Conference on Computer Vision ECCV 2024

AU - Goletto, Gabriele

AU - Nagarajan, Tushar

AU - Averta, Giuseppe

AU - Damen, Dima

PY - 2024/10/26

Y1 - 2024/10/26

N2 - Egocentric videos provide a unique perspective into individuals’ daily experiences, yet their unstructured nature presents challenges for perception. In this paper, we introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human’s ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate our understanding of very-long egocentric videos, we introduce the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. We showcase improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin.

AB - Egocentric videos provide a unique perspective into individuals’ daily experiences, yet their unstructured nature presents challenges for perception. In this paper, we introduce AMEGO, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. Inspired by the human’s ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate our understanding of very-long egocentric videos, we introduce the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. We showcase improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin.

U2 - 10.1007/978-3-031-72624-8_6

DO - 10.1007/978-3-031-72624-8_6

M3 - Conference Contribution (Conference Proceeding)

SN - 9783031726231

T3 - Lecture Notes in Computer Science

SP - 92

EP - 110

BT - Computer Vision – ECCV 2024

PB - Springer, Cham

Y2 - 29 September 2024 through 4 October 2024

ER -

AMEGO: Active Memory from long EGOcentric videos

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Handle.net

Fingerprint

Research output

Rescaling Egocentric Vision: Collection Pipeline and Challenges for EPIC-KITCHENS-100

Projects

8030 EPSRC via Oxford EP/T028572/1 Visual AI

8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship

Datasets

EPIC-KITCHENS VISOR

EPIC-KITCHENS-100

Cite this