Skip to main content
DORAS
DCU Online Research Access Service
Login (DCU Staff Only)
cushLEPOR uses LABSE distilled knowledge to improve correlation with human translation evaluations

Erofeev, Gleb, Sorokina, Irina, Han, Lifeng ORCID: 0000-0002-3221-2185 and Gladkoff, Serge (2021) cushLEPOR uses LABSE distilled knowledge to improve correlation with human translation evaluations. In: Machine Translation Summit 2021, 16-20 Aug 2021, USA (online). (In Press)

Full text available as:

[img]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
901kB

Abstract

Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python portable version we developed which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses LABSE distilled knowledge model to improve the metric agreement with human judgements by automatically optimised factor weights regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LABSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU (data available at \url{this https URL}).

Item Type:Conference or Workshop Item (Speech)
Event Type:Conference
Refereed:Yes
Additional Information:A presentation given in the Machine Translation Summit 2021 Conference, User Track.
Uncontrolled Keywords:Machine Translation Evaluation; Parameter Optimisation; Evaluation Metrics; Agreement; Statistical Analysis
Subjects:Computer Science > Algorithms
Computer Science > Artificial intelligence
Computer Science > Computational linguistics
Computer Science > Computer engineering
Computer Science > Information technology
Computer Science > Machine learning
Mathematics > Mathematical models
Mathematics > Statistics
DCU Faculties and Centres:DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing
Research Initiatives and Centres > ADAPT
Published in: Proceedings of MT Summit 2021 - User Track. . Association for Computational Linguistics.
Publisher:Association for Computational Linguistics
Official URL:https://amtaweb.org/mt-summit2021/
Copyright Information:© 2021 The Authors
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:ADAPT
ID Code:26182
Deposited On:07 Sep 2021 13:04 by Lifeng Han . Last Modified 07 Sep 2021 13:04

Downloads

Downloads per month over past year

Archive Staff Only: edit this record

Altmetric
- Altmetric
+ Altmetric
  • Student Email
  • Staff Email
  • Student Apps
  • Staff Apps
  • Loop
  • Disclaimer
  • Privacy
  • Contact Us