Erofeev, Gleb, Sorokina, Irina, Han, Lifeng ORCID: 0000-0002-3221-2185 and Gladkoff, Serge (2021) cushLEPOR uses LABSE distilled knowledge to improve correlation with human translation evaluations. In: Machine Translation Summit 2021, 16-20 Aug 2021, USA (online).
Abstract
Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python portable version we developed which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses LABSE distilled knowledge model to improve the metric agreement with human judgements by automatically optimised factor weights regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LABSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU (data available at \url{this https URL}).
Metadata
Item Type: | Conference or Workshop Item (Speech) |
---|---|
Event Type: | Conference |
Refereed: | Yes |
Uncontrolled Keywords: | Machine Translation Evaluation; Parameter Optimisation; Evaluation Metrics; Agreement; Statistical Analysis |
Subjects: | Computer Science > Algorithms Computer Science > Artificial intelligence Computer Science > Computational linguistics Computer Science > Computer engineering Computer Science > Information technology Computer Science > Machine learning Mathematics > Mathematical models Mathematics > Statistics |
DCU Faculties and Centres: | DCU Faculties and Schools > Faculty of Engineering and Computing > School of Computing Research Institutes and Centres > ADAPT |
Published in: | Proceedings of Machine Translation Summit XVIII: Users and Providers Track. . Association for Computational Linguistics. |
Publisher: | Association for Computational Linguistics |
Official URL: | https://aclanthology.org/2021.mtsummit-up.28 |
Copyright Information: | © 2021 The Authors |
Use License: | This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License |
Funders: | ADAPT |
ID Code: | 26182 |
Deposited On: | 07 Sep 2021 13:04 by Lifeng Han . Last Modified 16 Jan 2023 16:14 |
Documents
Full text available as:
Preview |
PDF
- Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution-Noncommercial 3.0 901kB |
Downloads
Downloads
Downloads per month over past year
Archive Staff Only: edit this record