Doherty, Stephen ORCID: 0000-0003-0887-1049, O'Brien, Sharon ORCID: 0000-0003-4864-5986 and Carl, Michael (2010) Eye tracking as an MT evaluation technique. Machine Translation, 24 (1). pp. 1-13. ISSN 0922-6567
Abstract
Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.
Metadata
Item Type: | Article (Published) |
---|---|
Refereed: | Yes |
Uncontrolled Keywords: | MT evaluation; Eye tracking; Gaze time; Fixation count; Fixation duration; Pupil dilation; HTER |
Subjects: | Computer Science > Machine translating Humanities > Linguistics |
DCU Faculties and Centres: | Research Institutes and Centres > Centre for Next Generation Localisation (CNGL) DCU Faculties and Schools > Faculty of Humanities and Social Science > School of Applied Language and Intercultural Studies |
Publisher: | Kluwer |
Official URL: | http://link.springer.com/article/10.1007/s10590-01... |
Copyright Information: | © 2010 Springer (Kluwer) The original publication is available at www.springerlink.com |
Use License: | This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License |
Funders: | Science Foundation Ireland |
ID Code: | 19471 |
Deposited On: | 02 Oct 2013 09:58 by Stephen Doherty . Last Modified 19 Jan 2022 12:28 |
Documents
Full text available as:
Preview |
PDF
- Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
226kB |
Metrics
Altmetric Badge
Dimensions Badge
Downloads
Downloads
Downloads per month over past year
Archive Staff Only: edit this record