On the same page? Comparing inter-annotator agreement in sentence
and document level human machine translation evaluation
Castilho, SheilaORCID: 0000-0002-8416-6555
(2020)
On the same page? Comparing inter-annotator agreement in sentence
and document level human machine translation evaluation.
In: Fifth Conference on Machine Translation, 19-20 Nov 2020, Dominican Republic (Online).
Document-level evaluation of machine translation has raised interest in the community especially since responses to the claims of “human parity” (Toral et al., 2018; L¨aubli et al.,2018) with document-level human evaluations have been published. Yet, little is known about best practices regarding human evaluation of machine translation at the documentlevel.
This paper presents a comparison of the differences in inter-annotator agreement between quality assessments using sentence and document-level set-ups. We report results of the agreement between professional translators for fluency and adequacy scales, error annotation, and pair-wise ranking, along with the effort needed to perform the different tasks. To best of our knowledge, this is the first study of its kind.
European Association for Machine Translation, Science Foundation Ireland Research Centres Programme (Grant 13/RC/2106) and is co-funded by the European Regional Development Fund.
ID Code:
25075
Deposited On:
12 Oct 2020 14:30 by
Sheila Castilho
. Last Modified 12 Jan 2021 12:13