Multimodal Speaker Diarization Utilizing Face Clustering Information

Ioannis Kapsouoras; Anastasios Tefas; Nikos Nikolaidis; Ioannis Pitas

doi:10.1007/978-3-319-21963-9_50

Multimodal Speaker Diarization Utilizing Face Clustering Information

Ioannis Kapsouoras, Anastasios Tefas, Nikos Nikolaidis, Ioannis Pitas

Department of Electrical & Electronic Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

317 Downloads (Pure)

Abstract

Multimodal clustering/diarization tries to answer the question ”who spoke when” by using audio and visual information. Diarization consists of two steps, at first segmentation of the audio information and detection of the speech segments and then clustering of the speech segments to group the speakers. This task has been mainly studied on audiovisual data from meetings, news broadcasts or talk shows. In this paper, we use visual information to aid speaker clustering. We tested the proposed method in three full length movies, i.e. a scenario much
more difficult than the ones used so far, where there is no certainty that speech segments and video appearances of actors will always overlap. The results proved that the visual information can improve the speaker clustering accuracy and hence the diarization process.

Original language	English
Title of host publication	Image and Graphics
Subtitle of host publication	8th International Conference, ICIG 2015, Tianjin, China, August 13-16, 2015, Proceedings, Part II
Editors	Yu-Jin Zhang
Publisher	Springer
Pages	547-554
Number of pages	8
ISBN (Electronic)	9783319219639
ISBN (Print)	9783319219622
DOIs	https://doi.org/10.1007/978-3-319-21963-9_50
Publication status	Published - 4 Aug 2015
Event	8th International Conference, ICIG 2015 - Tianjin, China Duration: 13 Aug 2015 → 16 Aug 2015

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	9218
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th International Conference, ICIG 2015
Country/Territory	China
City	Tianjin
Period	13/08/15 → 16/08/15

Keywords

Multiomodal
Diarization
Clustering
Movies

Access to Document

10.1007/978-3-319-21963-9_50

Ioannis Pitas - Multimodal speaker diarization utilizing face clustering information - 2015Accepted author manuscript, 113 KB

Handle.net

Persistent link

Cite this

Kapsouoras, I., Tefas, A., Nikolaidis, N., & Pitas, I. (2015). Multimodal Speaker Diarization Utilizing Face Clustering Information. In Y.-J. Zhang (Ed.), Image and Graphics: 8th International Conference, ICIG 2015, Tianjin, China, August 13-16, 2015, Proceedings, Part II (pp. 547-554). (Lecture Notes in Computer Science; Vol. 9218). Springer. https://doi.org/10.1007/978-3-319-21963-9_50

@inproceedings{e791e28fb76743a4a4e65d17e5debe57,

title = "Multimodal Speaker Diarization Utilizing Face Clustering Information",

abstract = "Multimodal clustering/diarization tries to answer the question ”who spoke when” by using audio and visual information. Diarization consists of two steps, at first segmentation of the audio information and detection of the speech segments and then clustering of the speech segments to group the speakers. This task has been mainly studied on audiovisual data from meetings, news broadcasts or talk shows. In this paper, we use visual information to aid speaker clustering. We tested the proposed method in three full length movies, i.e. a scenario muchmore difficult than the ones used so far, where there is no certainty that speech segments and video appearances of actors will always overlap. The results proved that the visual information can improve the speaker clustering accuracy and hence the diarization process.",

keywords = "Multiomodal, Diarization, Clustering, Movies",

author = "Ioannis Kapsouoras and Anastasios Tefas and Nikos Nikolaidis and Ioannis Pitas",

year = "2015",

month = aug,

day = "4",

doi = "10.1007/978-3-319-21963-9_50",

language = "English",

isbn = "9783319219622",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "547--554",

editor = "Yu-Jin Zhang",

booktitle = "Image and Graphics",

note = "8th International Conference, ICIG 2015 ; Conference date: 13-08-2015 Through 16-08-2015",

}

Kapsouoras, I, Tefas, A, Nikolaidis, N & Pitas, I 2015, Multimodal Speaker Diarization Utilizing Face Clustering Information. in Y-J Zhang (ed.), Image and Graphics: 8th International Conference, ICIG 2015, Tianjin, China, August 13-16, 2015, Proceedings, Part II. Lecture Notes in Computer Science, vol. 9218, Springer, pp. 547-554, 8th International Conference, ICIG 2015, Tianjin, China, 13/08/15. https://doi.org/10.1007/978-3-319-21963-9_50

Multimodal Speaker Diarization Utilizing Face Clustering Information. / Kapsouoras, Ioannis; Tefas, Anastasios; Nikolaidis, Nikos et al.
Image and Graphics: 8th International Conference, ICIG 2015, Tianjin, China, August 13-16, 2015, Proceedings, Part II. ed. / Yu-Jin Zhang. Springer, 2015. p. 547-554 (Lecture Notes in Computer Science; Vol. 9218).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Multimodal Speaker Diarization Utilizing Face Clustering Information

AU - Kapsouoras, Ioannis

AU - Tefas, Anastasios

AU - Nikolaidis, Nikos

AU - Pitas, Ioannis

PY - 2015/8/4

Y1 - 2015/8/4

N2 - Multimodal clustering/diarization tries to answer the question ”who spoke when” by using audio and visual information. Diarization consists of two steps, at first segmentation of the audio information and detection of the speech segments and then clustering of the speech segments to group the speakers. This task has been mainly studied on audiovisual data from meetings, news broadcasts or talk shows. In this paper, we use visual information to aid speaker clustering. We tested the proposed method in three full length movies, i.e. a scenario muchmore difficult than the ones used so far, where there is no certainty that speech segments and video appearances of actors will always overlap. The results proved that the visual information can improve the speaker clustering accuracy and hence the diarization process.

AB - Multimodal clustering/diarization tries to answer the question ”who spoke when” by using audio and visual information. Diarization consists of two steps, at first segmentation of the audio information and detection of the speech segments and then clustering of the speech segments to group the speakers. This task has been mainly studied on audiovisual data from meetings, news broadcasts or talk shows. In this paper, we use visual information to aid speaker clustering. We tested the proposed method in three full length movies, i.e. a scenario muchmore difficult than the ones used so far, where there is no certainty that speech segments and video appearances of actors will always overlap. The results proved that the visual information can improve the speaker clustering accuracy and hence the diarization process.

KW - Multiomodal

KW - Diarization

KW - Clustering

KW - Movies

U2 - 10.1007/978-3-319-21963-9_50

DO - 10.1007/978-3-319-21963-9_50

M3 - Conference Contribution (Conference Proceeding)

SN - 9783319219622

T3 - Lecture Notes in Computer Science

SP - 547

EP - 554

BT - Image and Graphics

A2 - Zhang, Yu-Jin

PB - Springer

T2 - 8th International Conference, ICIG 2015

Y2 - 13 August 2015 through 16 August 2015

ER -

Multimodal Speaker Diarization Utilizing Face Clustering Information

Abstract

Publication series

Conference

Keywords

Access to Document

Handle.net

Fingerprint

Cite this