The translator’s perspective on translation quality control processes for international large-scale assessment studies

Britta Upsing,Marc Rittberger

doi:10.12807/ti.110202.2018.a05

Abstract

International Large-Scale Assessment studies (iLSAs) like PISA (the Programme for International Student Assessment) and PIAAC (the Programme for the International Assessment of Adult Competencies) employ stringent quality control procedures for the translation of their test items. The translations are crucial: a test item should not become more or less difficult because of its translation, given that test results are used to assess and compare the competency levels of different populations across countries. This article discusses how PISA and PIAAC translation procedures have evolved from earlier translation quality processes, and the underlying assumptions about translation that have shaped this evolution. We then report on findings from a qualitative interview study with translators, reviewers, and translation managers who have been involved in PISA or PIAAC translation processes. The objective of the interview study is to analyse the quality control procedures from the perspective of translation players. How do translators prepare for and perform translations for iLSA studies, and how does this process compare with other translation assignments they receive? This comparison will give an understanding of whether translators, in general, believe the iLSA translation process and its quality control procedures provide adequate guidance for performing these translations. We finish by proposing recommendations for future iLSA translation processes.

Highlights

In 2006 a sudden shift in public opinion seemed to have occurred in Vietnam: one-third of respondents to an international survey favoured military rule when the previous survey had indicated 99 percent of the population was in favor (Kurzman, 2014)
In 2016, 20 in-depth qualitative interviews were conducted with translators, project managers, and reviewers with the goal of learning how translators compare translating for International Large-Scale Assessment studies (iLSAs) with other translation jobs they receive – and how these players perceive their work
The results presented pertain to four distinct parts of the translation process: Information and translation decisions, item-specific guidelines, feedback, and group reconciliations and meetings

Summary

Introduction

In 2006 a sudden shift in public opinion seemed to have occurred in Vietnam: one-third of respondents to an international survey favoured military rule when the previous survey had indicated 99 percent of the population was in favor (Kurzman, 2014). As Kurzman, a professor of sociology, argued in a Washington Post article, this shift had nothing to do with sociological or demographic changes in Vietnam: instead, he thinks that the 2006 survey had switched the wording of its Vietnamese translations. The validity of cross-national surveys or tests may be at stake when the translated versions of questionnaire or test items contain errors or (slight) deviations. The example posits that valid comparisons between countries (Vietnam and the United States, for instance) are only possible if the test instruments measure the same constructs across languages and countries.

Objectives

Methods

Results

Conclusion