This paper reports on a comparative evaluation of phrase-based statistical machine translation
(PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics,
including automatic evaluation as well as human rankings of adequacy and fluency, error-type
markup, and post-editing (technical and temporal) effort, performed by professional translators.
Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and
segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in
the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level
post-editing performance was not found to have significantly improved in NMT compared to
PBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create
a replicable semi-automated methodology for high-quality machine translation of educational
data.
This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:
TraMOOC project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No644333., ADAPT Centre for Digital Content Technology at Dublin City University is funded under the Science Foundation Ireland Research Centres Programme (Grant 13/RC/ 2106) and is co-funded under the European Regional Development Fund.
ID Code:
23083
Deposited On:
25 Apr 2019 15:30 by
Thomas Murtagh
. Last Modified 05 May 2023 16:28