Skip to main content
DORAS
DCU Online Research Access Service
Login (DCU Staff Only)
Results of the WMT17 metrics shared task

Bojar, Ondřej ORCID: 0000-0002-0606-0050, Graham, Yvette and Kamran, Amir (2017) Results of the WMT17 metrics shared task. In: Second Conference on Machine Translation (WMT17), 7-8 Sept 2017, Copenhagen, Denmark.

Full text available as:

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
988kB

Abstract

This paper presents the results of the WMT17 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT17 news translation task and Neural MT training task. We collected scores of 14 metrics from 8 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric’s scores correlate with WMT17 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in judging the quality of a particular sentence). This year, we build upon two types of manual judgements: direct assessment (DA) and HUME manual semantic judgements.

Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Machine translating
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of the Conference on Machine Translation (WMT), Shared Task Papers. 2. Association for Computational Linguistics (ACL).
Publisher:Association for Computational Linguistics (ACL)
Official URL:http://www.aclweb.org/anthology/W17-4755
Copyright Information:©2017 Association for Computational Linguistics
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Grants H2020-ICT-2014-1-645442 (QT21), H2020-ICT2014-1-644402 (HimL), Dutch organization for scientific research STW grant nr. 12271, ADAPT Centre for Digital Content Technology (www.adaptcentre.ie) at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund, Charles University Research Programme “Progres” Q18+Q48.
ID Code:23367
Deposited On:28 May 2019 10:14 by Thomas Murtagh . Last Modified 28 May 2019 10:14

Downloads

Downloads per month over past year

Archive Staff Only: edit this record

Altmetric
- Altmetric
+ Altmetric
  • Student Email
  • Staff Email
  • Student Apps
  • Staff Apps
  • Loop
  • Disclaimer
  • Privacy
  • Contact Us