Skip to content

Multimodality and the origin of a novel communication system in face-to-face interaction

Research output: Contribution to journalArticle

Original languageEnglish
Article number182056
Number of pages25
JournalRoyal Society Open Science
Volume7
Issue number1
DOIs
DateAccepted/In press - 27 Nov 2019
DatePublished (current) - 15 Jan 2020

Abstract

Face-to-face communication is multimodal at its core: it consists of a combination of vocal and visual signalling. However, current evidence suggests that, in the absence of an established communication system, visual signalling, especially in the form of visible gesture, is a more powerful form of communication than vocalization and therefore likely to have played a primary role in the emergence of human language. This argument is based on experimental evidence of how vocal and visual modalities (i.e. gesture) are employed to communicate about familiar concepts when participants cannot use their existing languages. To investigate this further, we introduce an experiment where pairs of participants performed a referential communication task in which they described unfamiliar stimuli in order to reduce reliance on conventional signals. Visual and auditory stimuli were described in three conditions: using visible gestures only, using non-linguistic vocalizations only and given the option to use both (multimodal communication). The results suggest that even in the absence of conventional signals, gesture is a more powerful mode of communication compared with vocalization, but that there are also advantages to multimodality compared to using gesture alone. Participants with an option to produce multimodal signals had comparable accuracy to those using only gesture, but gained an efficiency advantage. The analysis of the interactions between participants showed that interactants developed novel communication systems for unfamiliar stimuli by deploying different modalities flexibly to suit their needs and by taking advantage of multimodality when required.

    Research areas

  • language, evolution, multimodality, gesture, vocalization, interaction

Download statistics

No data available

Documents

Documents

  • Full-text PDF (final published version)

    Rights statement: This is the final published version of the article (version of record). It first appeared online via The Royal Society at https://royalsocietypublishing.org/doi/10.1098/rsos.182056. Please refer to any applicable terms of use of the publisher.

    Final published version, 1.31 MB, PDF document

    Licence: CC BY

DOI

View research connections

Related faculties, schools or groups