Document-level machine translation evaluation project: methodology,
effort and inter-annotator agreement
Castilho, SheilaORCID: 0000-0002-8416-6555
(2020)
Document-level machine translation evaluation project: methodology,
effort and inter-annotator agreement.
In: 22nd Annual Conference of the European Association for Machine Translation, 3-5 Nov 2020, Lisbon, Portugal (Online).
ISBN 978-989-33-0589-8
Recently, document-level (doc-level) human evaluation of machine translation
(MT) has raised interest in the community after a few attempts have disproved
claims of “human parity” (Toral et al.,
2018; Laubli et al., 2018). However, lit- ¨
tle is still known about best practices regarding doc-level human evaluation. This
project aims to identify methodologies to
better cope with i) the current state-of-theart (SOTA) human metrics, ii) a possible
complexity when assigning a single score
to a text consisted of ‘good’ and ‘bad’
sentences, iii) a possible tiredness bias in
doc-level set-ups, and iv) the difference in
inter-annotator agreement (IAA) between
sentence and doc-level set-ups.
Metadata
Item Type:
Conference or Workshop Item (Paper)
Event Type:
Conference
Refereed:
Yes
Uncontrolled Keywords:
machine translation evaluation; human evaluation of machine translation
This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:
EAMT under its programme “2019 Sponsorship of Activities, Science Foundation Ireland Research Centres Programme (Grant 13/RC/2106), European Regional Development Fund
ID Code:
24647
Deposited On:
18 Jun 2020 11:58 by
Sheila Castilho
. Last Modified 18 Jun 2020 12:09