How do measures of cognitive effort relate to each other? A multivariate analysis of post-editing process data

Research output: Contribution to journalArticle (Academic Journal)peer-review

5 Citations (Scopus)
290 Downloads (Pure)

Abstract

There has been growing interest of late in the cognitive effort required by post-editing of machine translation. Compared to number of editing operations, cognitive (or mental) effort is frequently considered a more decisive indicator of the overall effort expended by post-editors. Estimating cognitive effort is not straightforward, however. Previous studies often triangulate different measures to obtain a consensus, but little post-editing research to date has attempted to show how measures of cognitive effort relate to each other in a multivariate analysis. This paper addresses this by presenting an exploratory comparison of cognitive measures based on eye tracking, pauses, editing time, and subjective ratings collected in a post-editing task carried out by professional and non-professional participants. All measures correlated with each other, but a principal components analysis showed that the measures cluster together in different ways. In particular, measures that increase with task time alone behaved differently from the others, with higher mutual associations and higher reliability. Regarding differences between professional and non-professional participants, it was observed that subjective ratings were overall more strongly associated with objective measures in the case of professionals. Surprising findings from previous research based on pause ratio are discussed. The paper argues that a pause typology will benefit the study of pause lengths and cognitive effort in post-editing.
Original languageEnglish
Pages (from-to)41-62
Number of pages22
JournalMachine Translation
Volume30
Issue number1
DOIs
Publication statusPublished - 21 Jan 2017

Bibliographical note

E-pub ahead of print: 21 January 2017
Issue date: June 2016

Keywords

  • post-editing
  • machine translation
  • cognitive effort
  • eye tracking
  • pauses
  • subjective ratings

Fingerprint Dive into the research topics of 'How do measures of cognitive effort relate to each other? A multivariate analysis of post-editing process data'. Together they form a unique fingerprint.

Cite this