Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

Abstract

This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centered approach, the ISO/TR 16982 definition of usability—goal completion, satisfaction, effectiveness, and efficiency— is utilized and eye-tracking measures that are shown to be reliable indicators of cognitive effort are applied along with a posttask questionnaire. The study investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German, and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, the study found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.