Achieving accurate conclusions in evaluation of
automatic machine translation metrics
Graham, Yvette and Liu, QunORCID: 0000-0002-7000-1792
(2016)
Achieving accurate conclusions in evaluation of
automatic machine translation metrics.
In: 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 12-15 June 2016, San Diego, Ca. USA.
Automatic Machine Translation metrics, such
as BLEU, are widely used in empirical evaluation as a substitute for human assessment.
Subsequently, the performance of a given metric is measured by its strength of correlation
with human judgment. When a newly proposed metric achieves a stronger correlation
over that of a baseline, it is important to take
into account the uncertainty inherent in correlation point estimates prior to concluding
improvements in metric performance. Confidence intervals for correlations with human
judgment are rarely reported in metric evaluations, however, and when they have been
reported, the most suitable methods have unfortunately not been applied. For example,
incorrect assumptions about correlation sampling distributions made in past evaluations
risk over-estimation of significant differences
in metric performance. In this paper, we provide analysis of each of the issues that may
lead to inaccuracies before providing detail of
a method that overcomes previous challenges.
Additionally, we propose a new method of
translation sampling that in contrast achieves
genuine high conclusivity in evaluation of the
relative performance of metrics.
This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:
European Union Horizon 2020 research and innovation programme under grant agreement 645452 (QT21), ADAPT Centre for Digital Content Technology at Dublin City University funded under the SFI Research Centres Programme (Grant 13/RC/2106) co-funded under the European Regional Development Fund.
ID Code:
23208
Deposited On:
25 Apr 2019 13:57 by
Thomas Murtagh
. Last Modified 06 Mar 2020 09:47