Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

GaelEval: Benchmarking LLM Performance for Scottish Gaelic

Ó Meachair, Mícheál J. orcid logoORCID: 0000-0003-3931-5571, Devine, Peter, Lamb, Will, Alex, Beatrice, Ezeani, Ignatius, Knight, Dawn, Rayson, Paul and Wynne, Martin (2026) GaelEval: Benchmarking LLM Performance for Scottish Gaelic. In: LLMs4SSH (workshop co-located with LREC 2026), 11 May, 2026, Mallorca, Spain.

Abstract
Multilingual large language models (LLMs) often exhibit emergent 'shadow' capabilities in languages without official support, yet their performance on these languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority languages such as Scottish Gaelic, where translation benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally grounded translation benchmark and (iii) a large-scale cultural knowledge Q&A task. Evaluating 19 LLMs against a fluent-speaker human baseline (), we find that Gemini 3 Pro Preview achieves accuracy on the linguistic task, surpassing the human baseline (). Proprietary models consistently outperform open-weight systems, and in-language (Gaelic) prompting yields a small but stable advantage (+). On the cultural task, leading models exceed accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models.
Metadata
Item Type:Conference or Workshop Item (Paper)
Event Type:Conference
Refereed:Yes
Subjects:Computer Science > Artificial intelligence
Humanities > Language
DCU Faculties and Centres:UNSPECIFIED
Published in: Proceedings of LLMs4SSH (workshop co-located with LREC 2026). . Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities.
Publisher:Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities
Official URL:https://sites.google.com/view/llms4ssh-lrec2026
Copyright Information:Authors
ID Code:32914
Deposited On:06 Jul 2026 10:25 by Vidatum Academic . Last Modified 06 Jul 2026 10:25
Documents

Full text available as:

[thumbnail of GaelEval_2604.02135v1.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Creative Commons: Attribution 4.0
359kB
Metrics

Altmetric Badge

Dimensions Badge

Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record