It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Toby J Perrett; Tengda Han; Dima  Damen; Andrew Zisserman

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Toby J Perrett, Tengda Han, Dima Damen, Andrew Zisserman

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Abstract

Long videos contain many repeating actions, events and shots.
These repetitions are frequently given identical captions, which makes it
difficult to retrieve the exact desired clip using a text search. In this
paper, we formulate the problem of unique captioning: Given multiple
clips with the same caption, we generate a new caption for each clip
that uniquely identifies it. We propose Captioning by Discriminative
Prompting (CDP), which predicts a property that can separate identically
captioned clips, and use it to generate unique captions. We introduce
two benchmarks for unique captioning, based on egocentric footage
and timeloop movies – where repeating actions are common. We demonstrate
that captions generated by CDP improve text-to-video R@1 by
15% for egocentric videos and 10% in timeloop movies.

Original language	English
Title of host publication	Computer Vision – ACCV 2024
Subtitle of host publication	17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings
Publisher	Springer, Singapore
Publication status	Accepted/In press - 11 Oct 2024
Event	Asian Conference on Computer Vision - Hanoi, Viet Nam Duration: 8 Dec 2024 → 12 Dec 2024 https://accv2024.org

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	15480
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Asian Conference on Computer Vision
Abbreviated title	ACCV
Country/Territory	Viet Nam
City	Hanoi
Period	8/12/24 → 12/12/24
Internet address	https://accv2024.org

Handle.net

Persistent link

Cite this

@inproceedings{c8f8f21acb8a4ac3ad38942f21dac9b0,

title = "It{\textquoteright}s Just Another Day: Unique Video Captioning by Discriminitive Prompting",

abstract = "Long videos contain many repeating actions, events and shots.These repetitions are frequently given identical captions, which makes itdifficult to retrieve the exact desired clip using a text search. In thispaper, we formulate the problem of unique captioning: Given multipleclips with the same caption, we generate a new caption for each clipthat uniquely identifies it. We propose Captioning by DiscriminativePrompting (CDP), which predicts a property that can separate identicallycaptioned clips, and use it to generate unique captions. We introducetwo benchmarks for unique captioning, based on egocentric footageand timeloop movies – where repeating actions are common. We demonstratethat captions generated by CDP improve text-to-video R@1 by15% for egocentric videos and 10% in timeloop movies.",

author = "Perrett, {Toby J} and Tengda Han and Dima Damen and Andrew Zisserman",

year = "2024",

month = oct,

day = "11",

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Singapore",

booktitle = "Computer Vision – ACCV 2024",

address = "Singapore",

note = "Asian Conference on Computer Vision, ACCV ; Conference date: 08-12-2024 Through 12-12-2024",

url = "https://accv2024.org",

}

Perrett, TJ, Han, T, Damen, D & Zisserman, A 2024, It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. in Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. Lecture Notes in Computer Science, vol. 15480, Springer, Singapore, Asian Conference on Computer Vision, Hanoi, Viet Nam, 8/12/24.

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting. / Perrett, Toby J; Han, Tengda; Damen, Dima et al.
Computer Vision – ACCV 2024: 17th Asian Conference on Computer Vision, Hanoi, Vietnam, December 8–12, 2024, Proceedings. Springer, Singapore, 2024. (Lecture Notes in Computer Science; Vol. 15480).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - It’s Just Another Day

T2 - Asian Conference on Computer Vision

AU - Perrett, Toby J

AU - Han, Tengda

AU - Damen, Dima

AU - Zisserman, Andrew

PY - 2024/10/11

Y1 - 2024/10/11

N2 - Long videos contain many repeating actions, events and shots.These repetitions are frequently given identical captions, which makes itdifficult to retrieve the exact desired clip using a text search. In thispaper, we formulate the problem of unique captioning: Given multipleclips with the same caption, we generate a new caption for each clipthat uniquely identifies it. We propose Captioning by DiscriminativePrompting (CDP), which predicts a property that can separate identicallycaptioned clips, and use it to generate unique captions. We introducetwo benchmarks for unique captioning, based on egocentric footageand timeloop movies – where repeating actions are common. We demonstratethat captions generated by CDP improve text-to-video R@1 by15% for egocentric videos and 10% in timeloop movies.

AB - Long videos contain many repeating actions, events and shots.These repetitions are frequently given identical captions, which makes itdifficult to retrieve the exact desired clip using a text search. In thispaper, we formulate the problem of unique captioning: Given multipleclips with the same caption, we generate a new caption for each clipthat uniquely identifies it. We propose Captioning by DiscriminativePrompting (CDP), which predicts a property that can separate identicallycaptioned clips, and use it to generate unique captions. We introducetwo benchmarks for unique captioning, based on egocentric footageand timeloop movies – where repeating actions are common. We demonstratethat captions generated by CDP improve text-to-video R@1 by15% for egocentric videos and 10% in timeloop movies.

UR - https://accv2024.org/

M3 - Conference Contribution (Conference Proceeding)

T3 - Lecture Notes in Computer Science

BT - Computer Vision – ACCV 2024

PB - Springer, Singapore

Y2 - 8 December 2024 through 12 December 2024

ER -

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Abstract

Publication series

Conference

Handle.net

Other files and links

Embargoed Document

Visual AI - Full Programme Grant Extension

8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship

Cite this

It’s Just Another Day: Unique Video Captioning by Discriminitive Prompting

Abstract

Publication series

Conference

Handle.net

Other files and links

Embargoed Document

Projects

Visual AI - Full Programme Grant Extension

8459 EPSRC EP/T004991/1 UMPIRE - Dima Aldamen Fellowship

Cite this