Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation: Revisiting Research Conventions

Talia Isaacs, R.I Thomson

Research output: Contribution to journalArticle (Academic Journal)peer-review

73 Citations (Scopus)
184 Downloads (Pure)


This mixed-methods study examines the effects of rating scale length and rater experience on listeners' judgments of second-language (L2) speech. Twenty experienced and 20 novice raters, who were randomly assigned to 5-point or 9-point rating scale conditions, judged speech samples of 38 newcomers to Canada on numerical rating scales for comprehensibility, accentedness, and fluency. Results yielded high Cronbach's alpha coefficients and no group differences for rating scale length or rater experience. However, Rasch category probability plots revealed that raters had difficulty differentiating between scale steps, particularly in mid-scale range—a challenge that was exacerbated in the 9-point scale condition. Evidence from verbal protocols and posttask interviews suggested that experienced and novice raters adopted strategies to either draw on or offset their perceived experience with L2 speech in conducting their ratings. Implications for L2 pronunciation research are discussed, including the need to draw on raters' perceptions to develop a greater understanding of major listener perceptual constructs in L2 pronunciation and to move beyond adherence to the native speaking standard in data collection procedures.
Translated title of the contributionRater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions
Original languageEnglish
Pages (from-to)135-159
Number of pages25
JournalLanguage Assessment Quarterly
Issue number2
Publication statusPublished - 22 May 2013

Fingerprint Dive into the research topics of 'Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation: Revisiting Research Conventions'. Together they form a unique fingerprint.

Cite this