Utilizing AI questionnaire translations in cross-cultural and intercultural research: Insights and recommendations

Abstract

In this research paper, we investigated the viability of AI-supported translations of survey materials in intercultural and cross-cultural research, comparing the quality of machine translations to traditional human translations. Focusing on the HEXACO personality inventory, we translated the original English inventory using Google Translate and GPT-3.5 into 33 languages for which validated human translations exist. The statistical similarity between human- and machine-generated translations varied considerably between the target languages. It was highest for target languages from the same language family as the source language, arguably because this relatedness allowed for more direct machine translations. Consistent with this reasoning, the genetic similarity between languages largely explained the differences observed. GPT’s temperature setting determining how stringently or freely a text is translated had little influence on the similarity estimates, but very high levels tended to produce somewhat lower statistical similarity. To validate the quality of the machine translations, a group of social scientists rated the translation in a language for which the human and machine translations statistically converged strongly. Although the human translation was rated as being of higher quality than four out of five machine translations, these differences were relatively small. Crucially, the social scientists did not rate the human translation as significantly better than the GPT 3.5 translation with the lowest temperature setting. Based on these insights, we propose a framework outlining four recommendations for utilizing AI-supported translation in cross-cultural and intercultural research, involving AI to varying degrees in the forward-back translation process.