Disentangling the properties of human evaluation methods: a classification system to support comparability, meta-evaluation and reproducibility testing
Belz, AnyaORCID: 0000-0002-0552-8096, Mille, SimonORCID: 0000-0002-8852-2764 and Howcroft, David M.ORCID: 0000-0002-0810-9065
(2020)
Disentangling the properties of human evaluation methods: a classification system to support comparability, meta-evaluation and reproducibility testing.
In: 13th International Natural Language Generation Conference 2020 (INLG'20), 15-18 Dec 2020, Dublin, Ireland.
Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs. This has serious implications for reproducibility testing and meta-evaluation, in particular given that human evaluation is considered the gold standard against which the trustworthiness of automatic metrics is gauged. %and merging others, as well as deciding which evaluations should be able to reproduce each other’s results. Using examples from NLG, we propose a classification system for evaluations based on disentangling (i) what is being evaluated (which aspect of quality), and (ii) how it is evaluated in specific (a) evaluation modes and (b) experimental designs. We show that this approach provides a basis for determining comparability, hence for comparison of evaluations across papers, meta-evaluation experiments, reproducibility testing.
Davis, Brian, Graham, Yvette and Kelleher, John D., (eds.)
Proceedings of the 13th International Conference on Natural Language Generation.
.
Association for Computational Linguistics (ACL).