Data-to-text generation for severely under-resourced languages with GPT-3.5: a bit of help needed from Google Translate
Lorandi, MichelaORCID: 0000-0002-6131-8763 and Belz, AnyaORCID: 0000-0002-0552-8096
(2023)
Data-to-text generation for severely under-resourced languages with GPT-3.5: a bit of help needed from Google Translate.
In: 16th International Natural Language Generation Conference - Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge, 11-12 Sep 2023, Prague, Czech Republic.
(In Press)
LLMs like GPT are great at tasks involving English which dominates in their training data. In this paper, we look at how they cope with tasks involving languages that are severely under-represented in their training data, in the context of data-to-text generation for Irish, Maltese, Welsh and Breton. During the prompt-engineering phase we tested a range of prompt types and formats on GPT-3.5 and~4 with a small sample of example input/output pairs. We then fully evaluated the two most promising prompts in two scenarios: (i) direct generation into the under-resourced language, and (ii) generation into English followed by translation into the under-resourced language. We find that few-shot prompting works better for direct generation into under-resourced languages, but that the difference disappears when pivoting via English. The few-shot + translation system variants were submitted to the WebNLG 2023 shared task where they outperformed competitor systems by substantial margins in all languages on all metrics. We conclude that good performance on under-resourced languages can be achieved out-of-the box with state-of-the-art LLMs. However, our best results (for Welsh) remain well below the lowest ranked English system at WebNLG'20.
DCU-NLG Research Group at DCU, Science Foundation Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No.18/CRT/6224, Science Foundation Ireland under Grant Agreement No.13/RC/2106_P2 at the ADAPT SFI Research Centre at Dublin City University
ID Code:
28947
Deposited On:
20 Sep 2023 14:03 by
Michela Lorandi
. Last Modified 20 Sep 2023 14:03