Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels

Biao Zeng; Rui Wang; Guoxing Yu; Christian Dobel

doi:10.21437/Interspeech.2021-1741

Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels

Biao Zeng, Rui Wang, Guoxing Yu, Christian Dobel

School of Education

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Abstract

This study investigated Chinese speakers’ eye movements when they were asked to identify audiovisual Mandarin lexical tones and vowels. In the lexical tone identification task, Chinese speakers were presented with an audiovisual clip of Mandarin monosyllables (/ă/, /à/, /ĭ/, /ì/) and asked to identify whether the syllables were presented in a dipping (/ă/, / ĭ/) or falling tone (/à/, /ì/). In the vowel identification task, they were asked to identify whether the vowels were/a/or/i/regardless of lexical tone. These audiovisual syllables were presented in clear, noisy, and silent conditions. An eye-tracker recorded the participants’ eye movements. Results showed participants gazed more at the mouth than the eyes in both lexical tones and vowels. Additionally, when acoustic conditions degraded from clear to noisy and eventually silent, Chinese speakers increased their gaze towards the mouth rather than the eyes. These findings suggest the mouth to be the primary area that is utilised during audiovisual speech perception. The similar patterns of eye movements between vowels and lexical tones indicate that the mouth acts as a perceptual cue that provides articulatory information.

Original language	English
Title of host publication	22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
Publisher	International Speech Communication Association
Pages	2903-2906
Number of pages	4
ISBN (Electronic)	9781713836902
DOIs	https://doi.org/10.21437/Interspeech.2021-1741
Publication status	Published - 2021
Event	22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021 - Brno, Czech Republic Duration: 30 Aug 2021 → 3 Sept 2021

Publication series

Name	Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Volume	4
ISSN (Print)	2308-457X
ISSN (Electronic)	1990-9772

Conference

Conference	22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
Country/Territory	Czech Republic
City	Brno
Period	30/08/21 → 3/09/21

Bibliographical note

Funding Information:
The study was funded by the British Academy Small Grant (SG152162). We also acknowledge Bournemouth University for use of their research facility and Ms. Bridie Stone and Mr. Josh Molina for assisting in the proofreading and three anonymous reviewers.

Publisher Copyright:
Copyright © 2021 ISCA.

Keywords

Audiovisual speech
Chinese
Eye movement
Gaze
Lexical tone
Vowel

Access to Document

10.21437/Interspeech.2021-1741

Handle.net

Persistent link

Cite this

Zeng, B., Wang, R., Yu, G., & Dobel, C. (2021). Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021 (pp. 2903-2906). (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH; Vol. 4). International Speech Communication Association. https://doi.org/10.21437/Interspeech.2021-1741

Zeng, Biao ; Wang, Rui ; Yu, Guoxing et al. / Primacy of mouth over eyes : Eye movement evidence from audiovisual Mandarin lexical tones and vowels. 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021. International Speech Communication Association, 2021. pp. 2903-2906 (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH).

@inproceedings{2a883e885c834611b5e0db9dc8a50422,

title = "Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels",

abstract = "This study investigated Chinese speakers{\textquoteright} eye movements when they were asked to identify audiovisual Mandarin lexical tones and vowels. In the lexical tone identification task, Chinese speakers were presented with an audiovisual clip of Mandarin monosyllables (/{\u a}/, /{\`a}/, /ĭ/, /{\`i}/) and asked to identify whether the syllables were presented in a dipping (/{\u a}/, / ĭ/) or falling tone (/{\`a}/, /{\`i}/). In the vowel identification task, they were asked to identify whether the vowels were/a/or/i/regardless of lexical tone. These audiovisual syllables were presented in clear, noisy, and silent conditions. An eye-tracker recorded the participants{\textquoteright} eye movements. Results showed participants gazed more at the mouth than the eyes in both lexical tones and vowels. Additionally, when acoustic conditions degraded from clear to noisy and eventually silent, Chinese speakers increased their gaze towards the mouth rather than the eyes. These findings suggest the mouth to be the primary area that is utilised during audiovisual speech perception. The similar patterns of eye movements between vowels and lexical tones indicate that the mouth acts as a perceptual cue that provides articulatory information.",

keywords = "Audiovisual speech, Chinese, Eye movement, Gaze, Lexical tone, Vowel",

author = "Biao Zeng and Rui Wang and Guoxing Yu and Christian Dobel",

note = "Funding Information: The study was funded by the British Academy Small Grant (SG152162). We also acknowledge Bournemouth University for use of their research facility and Ms. Bridie Stone and Mr. Josh Molina for assisting in the proofreading and three anonymous reviewers. Publisher Copyright: Copyright {\textcopyright} 2021 ISCA.; 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021 ; Conference date: 30-08-2021 Through 03-09-2021",

year = "2021",

doi = "10.21437/Interspeech.2021-1741",

language = "English",

series = "Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH",

publisher = "International Speech Communication Association",

pages = "2903--2906",

booktitle = "22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021",

}

Zeng, B, Wang, R, Yu, G & Dobel, C 2021, Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels. in 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 4, International Speech Communication Association, pp. 2903-2906, 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, Brno, Czech Republic, 30/08/21. https://doi.org/10.21437/Interspeech.2021-1741

Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels. / Zeng, Biao; Wang, Rui; Yu, Guoxing et al.
22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021. International Speech Communication Association, 2021. p. 2903-2906 (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH; Vol. 4).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Primacy of mouth over eyes

T2 - 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021

AU - Zeng, Biao

AU - Wang, Rui

AU - Yu, Guoxing

AU - Dobel, Christian

N1 - Funding Information: The study was funded by the British Academy Small Grant (SG152162). We also acknowledge Bournemouth University for use of their research facility and Ms. Bridie Stone and Mr. Josh Molina for assisting in the proofreading and three anonymous reviewers. Publisher Copyright: Copyright © 2021 ISCA.

PY - 2021

Y1 - 2021

N2 - This study investigated Chinese speakers’ eye movements when they were asked to identify audiovisual Mandarin lexical tones and vowels. In the lexical tone identification task, Chinese speakers were presented with an audiovisual clip of Mandarin monosyllables (/ă/, /à/, /ĭ/, /ì/) and asked to identify whether the syllables were presented in a dipping (/ă/, / ĭ/) or falling tone (/à/, /ì/). In the vowel identification task, they were asked to identify whether the vowels were/a/or/i/regardless of lexical tone. These audiovisual syllables were presented in clear, noisy, and silent conditions. An eye-tracker recorded the participants’ eye movements. Results showed participants gazed more at the mouth than the eyes in both lexical tones and vowels. Additionally, when acoustic conditions degraded from clear to noisy and eventually silent, Chinese speakers increased their gaze towards the mouth rather than the eyes. These findings suggest the mouth to be the primary area that is utilised during audiovisual speech perception. The similar patterns of eye movements between vowels and lexical tones indicate that the mouth acts as a perceptual cue that provides articulatory information.

AB - This study investigated Chinese speakers’ eye movements when they were asked to identify audiovisual Mandarin lexical tones and vowels. In the lexical tone identification task, Chinese speakers were presented with an audiovisual clip of Mandarin monosyllables (/ă/, /à/, /ĭ/, /ì/) and asked to identify whether the syllables were presented in a dipping (/ă/, / ĭ/) or falling tone (/à/, /ì/). In the vowel identification task, they were asked to identify whether the vowels were/a/or/i/regardless of lexical tone. These audiovisual syllables were presented in clear, noisy, and silent conditions. An eye-tracker recorded the participants’ eye movements. Results showed participants gazed more at the mouth than the eyes in both lexical tones and vowels. Additionally, when acoustic conditions degraded from clear to noisy and eventually silent, Chinese speakers increased their gaze towards the mouth rather than the eyes. These findings suggest the mouth to be the primary area that is utilised during audiovisual speech perception. The similar patterns of eye movements between vowels and lexical tones indicate that the mouth acts as a perceptual cue that provides articulatory information.

KW - Audiovisual speech

KW - Chinese

KW - Eye movement

KW - Gaze

KW - Lexical tone

KW - Vowel

UR - http://www.scopus.com/inward/record.url?scp=85119215279&partnerID=8YFLogxK

U2 - 10.21437/Interspeech.2021-1741

DO - 10.21437/Interspeech.2021-1741

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85119215279

T3 - Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

SP - 2903

EP - 2906

BT - 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021

PB - International Speech Communication Association

Y2 - 30 August 2021 through 3 September 2021

ER -

Zeng B, Wang R, Yu G, Dobel C. Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021. International Speech Communication Association. 2021. p. 2903-2906. (Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH). doi: 10.21437/Interspeech.2021-1741

Primacy of mouth over eyes: Eye movement evidence from audiovisual Mandarin lexical tones and vowels

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this