This paper reports results from a reproduction study in which we repeated the human evaluation of the PASS Dutch-language football report generation system (van der Lee et al., 2017). The work was carried out as part of the ReproGen Shared Task on Reproducibility of Human Evaluations in NLG, in Track A (Paper 1). We aimed to repeat the original study exactly, with the main difference that a different set of evaluators was used. We describe the study design, present the results from the original and the reproduction study, and then compare and analyse the differences between the two sets of results. For the two ‘headline’ results of average Fluency and Clarity, we find that in both studies, the system was rated more highly for Clarity than for Fluency, and Clarity had higher standard deviation. Clarity and Fluency ratings were higher, and their standard deviations lower, in the reproduction study than in the original study by substantial margins. Clarity had a higher degree of reproducibility than Fluency, as measured by the coefficient of variation. Data and code are publicly available.
Belz, Anya, Fan, Angela, Reiter, Ehud and Sripada, Yaji, (eds.)
Procceedings of the 14th International Conference on Natural Language Generation (INLG 2021).
1.
Association for Computational Linguistics (ACL).
European Commission under the H2020 program contract numbers 786731, 825079, 870930 and 952133, Science Foundation Ireland through the SFI Research Centres Programme, and co-funded under the European Regional Development Fund (ERDF) Grant 13/RC/2106., Brazilian agency CAPES under Post-doctoral grant No. 88887.508597/2020-00
ID Code:
28637
Deposited On:
06 Jul 2023 15:16 by
Anya Belz
. Last Modified 10 Jul 2023 08:30