Challenges of a cross-national computer-based test adaptation

Abstract

In an increasingly globalized world, the call for internationally comparable competence measurements has emerged. After several international studies on pre-college education, the focus has shifted to international assessments of vocational education and training (VET). VET researchers in Germany developed a computer-based test (ALUSIM) that measures the competence of German apprentices in internationally defined core commercial areas. Our own study deals with the adaptation of this test to Switzerland and a discussion of the challenges involved, with the aim of providing guidance for future adaptations. More specifically, despite commonalities between the German and Swiss VET systems, it is necessary to examine whether the contents and technical aspects of ALUSIM are appropriate for Swiss conditions in order to ensure validity and evidence based on test content. Several methods were used to examine the criteria context, construct, and IT components of the German computer-based test ALUSIM in order to successfully adapt it for Swiss commercial VET. To this end we first analyzed and compared the German and Swiss commercial VET systems and commercial curricula (context) to assess whether the tasks of the test were also embedded in the Swiss curriculum and whether any specific Swiss commercial contents were not represented by the test. Second, we interviewed experts in the commercial area to learn more about representative commercial job requirements (construct). Finally, we interviewed apprentices and tested our initial adaptation of ALUSIM to the Swiss context in order to assess the test’s IT requirements. The analysis revealed similarities between the German and Swiss VET and the construct ‘commercial competence’. However, commercial work conditions and cultural characteristics differ between these countries and lead to different job requirements. Therefore, only a subset of ALUSIM tasks is valid for the construct ‘commercial competence’ in Switzerland. Thus, the addition of further tasks for Switzerland would more validly represent the construct ‘commercial competence’. Moreover, IT components need to be adjusted because the technical implementation represents the measured construct. The proceedings and findings of our adaptation study imply that context, construct, and IT components need to be analyzed before doing adaptations. Even when dealing with countries with a similar context such as comparable educational system and language, it is necessary to carefully examine and test an adaptation in advance. Therefore, creating a successful and internationally comparable adaptation is admittedly possible but challenging, costly and time-consuming.