Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Results of the WMT17 metrics shared task

Bojar, Ondřej orcid logoORCID: 0000-0002-0606-0050, Graham, Yvette and Kamran, Amir (2017) Results of the WMT17 metrics shared task. In: Second Conference on Machine Translation (WMT17), 7-8 Sept 2017, Copenhagen, Denmark.

Abstract
This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT17 news translation task and Neural MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric’s scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a particular sentence). This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judgements.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Machine translating
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Institutes and Centres > ADAPT
Published in: Proceedings of the Conference on Machine Translation (WMT), Shared Task Papers. 2. Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:http://www.aclweb.org/anthology/W17-4755
Copyright Information:©2017 Association for Computational Linguistics
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Grants H2020-ICT-2014-1-645442 (QT21), H2020-ICT2014-1-644402 (HimL), Dutch organization for scientific research STW grant nr. 12271, ADAPT Centre for Digital Content Technology (www.adaptcentre.ie) at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund, Charles University Research Programme “Progres” Q18+Q48.
ID Code:23367
Deposited On:28 May 2019 10:14 by Thomas Murtagh . Last Modified 28 May 2019 10:14
Documents

Full text available as:

[thumbnail of Results_of_the_WMT17_Metrics_Shared_Task[1].pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
988kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record