It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Toby J Perrett; Tengda Han; Dima  Damen; Andrew Zisserman

doi:10.1007/978-981-96-0908-6_16

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Toby J Perrett^*
, Tengda Han
, Dima Damen
, Andrew Zisserman

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

19 Downloads (Pure)

Abstract

Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies – where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.

Original language	English
Title of host publication	Computer Vision – ACCV 2024
Subtitle of host publication	17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings
Editors	Minsu Cho, Ivan Laptev, Du Tran, Angela Yao, Hongbin Zha
Publisher	Springer, Singapore
Chapter	16
Pages	275-293
Number of pages	19
Volume	3
ISBN (Electronic)	9789819609086
ISBN (Print)	9789819609079
DOIs	https://doi.org/10.1007/978-981-96-0908-6_16
Publication status	Published - 7 Dec 2024
Event	Asian Conference on Computer Vision - Hanoi, Viet Nam Duration: 8 Dec 2024 → 12 Dec 2024 https://accv2024.org

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15474 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Asian Conference on Computer Vision
Abbreviated title	ACCV
Country/Territory	Viet Nam
City	Hanoi
Period	8/12/24 → 12/12/24
Internet address	https://accv2024.org

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

Access to Document

10.1007/978-981-96-0908-6_16Licence: Unspecified

Full-text PDF (accepted manuscript)
This is the accepted author manuscript (AAM) of the article which has been made Open Access under the University of Bristol's Scholarly Works Policy. The final published version (Version of Record) can be found on the publisher's website. The copyright of any third-party content, such as images, remains with the copyright holder.
Accepted author manuscript, 11.3 MBLicence: CC BY

Handle.net

Persistent link

Cite this

Perrett, T. J., Han, T., Damen, D., & Zisserman, A. (2024). It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. In M. Cho, I. Laptev, D. Tran, A. Yao, & H. Zha (Eds.), Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings (Vol. 3, pp. 275-293). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15474 LNCS). Springer, Singapore. https://doi.org/10.1007/978-981-96-0908-6_16

Perrett, Toby J ; Han, Tengda ; Damen, Dima et al. / It’s Just Another Day : Unique Video Captioning by Discriminitive Prompting. Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. editor / Minsu Cho ; Ivan Laptev ; Du Tran ; Angela Yao ; Hongbin Zha. Vol. 3 Springer, Singapore, 2024. pp. 275-293 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c8f8f21acb8a4ac3ad38942f21dac9b0,

title = "It{\textquoteright}s Just Another Day: Unique Video Captioning by Discriminitive Prompting",

abstract = "Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies – where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15\% for egocentric videos and 10\% in timeloop movies.",

author = "Perrett, \{Toby J\} and Tengda Han and Dima Damen and Andrew Zisserman",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.; Asian Conference on Computer Vision, ACCV ; Conference date: 08-12-2024 Through 12-12-2024",

year = "2024",

month = dec,

day = "7",

doi = "10.1007/978-981-96-0908-6\_16",

language = "English",

isbn = "9789819609079",

volume = "3",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer, Singapore",

pages = "275--293",

editor = "Minsu Cho and Ivan Laptev and Du Tran and Angela Yao and Hongbin Zha",

booktitle = "Computer Vision – ACCV 2024",

address = "Singapore",

url = "https://accv2024.org",

}

Perrett, TJ, Han, T, Damen, D & Zisserman, A 2024, It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. in M Cho, I Laptev, D Tran, A Yao & H Zha (eds), Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. vol. 3, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15474 LNCS, Springer, Singapore, pp. 275-293, Asian Conference on Computer Vision, Hanoi, Viet Nam, 8/12/24. https://doi.org/10.1007/978-981-96-0908-6_16

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. / Perrett, Toby J; Han, Tengda; Damen, Dima et al.
Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. ed. / Minsu Cho; Ivan Laptev; Du Tran; Angela Yao; Hongbin Zha. Vol. 3 Springer, Singapore, 2024. p. 275-293 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15474 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - It’s Just Another Day

T2 - Asian Conference on Computer Vision

AU - Perrett, Toby J

AU - Han, Tengda

AU - Damen, Dima

AU - Zisserman, Andrew

PY - 2024/12/7

Y1 - 2024/12/7

N2 - Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies – where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.

AB - Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies – where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.

UR - https://accv2024.org/

U2 - 10.1007/978-981-96-0908-6_16

DO - 10.1007/978-981-96-0908-6_16

M3 - Conference Contribution (Conference Proceeding)

SN - 9789819609079

VL - 3

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 275

EP - 293

BT - Computer Vision – ACCV 2024

A2 - Cho, Minsu

A2 - Laptev, Ivan

A2 - Tran, Du

A2 - Yao, Angela

A2 - Zha, Hongbin

PB - Springer, Singapore

Y2 - 8 December 2024 through 12 December 2024

ER -

Perrett TJ, Han T, Damen D, Zisserman A. It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. In Cho M, Laptev I, Tran D, Yao A, Zha H, editors, Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. Vol. 3. Springer, Singapore. 2024. p. 275-293. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-981-96-0908-6_16

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Handle.net

Other files and links

Visual AI - Full Programme Grant Extension

8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship

Best Paper Award - ACCV 2024

Cite this

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Handle.net

Other files and links

Projects

Visual AI - Full Programme Grant Extension

8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship

Prizes

Best Paper Award - ACCV 2024

Cite this