Assessing the usability of raw machine translation output: A user-centered study using eye tracking
Doherty, StephenORCID: 0000-0003-0887-1049 and O'Brien, SharonORCID: 0000-0003-4864-5986
(2013)
Assessing the usability of raw machine translation output: A user-centered study using eye tracking.
International Journal of Human Computer Interaction, 30
(1).
pp. 40-51.
ISSN 1532-7590
This paper reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centred approach, we utilize the ISO/TR 16982 definition of usability - goal completion, satisfaction, effectiveness, and efficiency – and apply eye-tracking measures shown to be reliable indicators of cognitive effort, along with a post-task questionnaire. We investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, we found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw
machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.