EMMA: A Novel Evaluation Metric for Morphological Analysis

Sebastian R Spiegler, Christian Monson

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

18 Citations (Scopus)

Abstract

We present a novel Evaluation Metric for Morphological Analysis (EMMA) that is both linguistically appealing and empirically sound. EMMA uses a graph-based assignment algorithm, optimized via integer linear programming, to match morphemes of predicted word analyses to the analyses of a morphologically rich answer key. This is necessary especially for unsupervised morphology analysis systems which do not have access to linguistically motivated morpheme labels. Across 3 languages, EMMA scores of 14 systems have a substantially greater positive correlation with mean average precision in an information retrieval (IR) task than do scores from the metric currently used by the Morpho Challenge (MC) competition series. We compute EMMA and MC metric scores for 93 separate system-language pairs from the 2007, 2008, and 2009 MC competitions, demonstrating that EMMA is not susceptible to two types of gaming that have plagued recent MC competitions: Ambiguity Hijacking and Shared Morpheme Padding. The EMMA evaluation script is publicly available from http://www.cs.bris.ac.uk/Research/MachineLearning/Morphology/Resources/.
Translated title of the contributionEMMA: A Novel Evaluation Metric for Morphological Analysis
Original languageEnglish
Title of host publicationProceedings of the 23rd International Conference on Computational Linguistics (COLING)
Publication statusPublished - 2010

Bibliographical note

Other page information: -
Conference Proceedings/Title of Journal: Proceedings of the 23rd International Conference on Computational Linguistics (COLING)
Other identifier: 2001226

Fingerprint

Dive into the research topics of 'EMMA: A Novel Evaluation Metric for Morphological Analysis'. Together they form a unique fingerprint.

Cite this