A survey of recent error annotation schemes for automatically generated text
Huidrom, RudaliORCID: 0000-0003-0630-3603 and Belz, AnyaORCID: 0000-0002-0552-8096
(2022)
A survey of recent error annotation schemes for automatically generated text.
In: 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), 7 Dec 2022, Abu Dhabi, UAE and Online.
While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error analysis is receiving increasing attention, with numerous error annotation schemes being proposed for automatically generated text. However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity. In this paper, our aim is to map out recent work on annotating errors in automatically generated text, with a particular focus on error taxonomies. We describe our systematic paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them. Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.
ADAPT Centre for Digital Media Technology funded by Science Foundation Ireland SFI Research Centres Programme and is co-funded under the European Regional Development Fund (ERDF) Grant 13/RC/2106.
ID Code:
28660
Deposited On:
04 Jul 2023 14:51 by
Anya Belz
. Last Modified 12 Jul 2023 11:12