Test Difficulty in Second Language Setting: Measuring With Receiver Operating Characteristic

Abstract

In order to assess the accuracy and validity of proficiency diagnostic tests in Second Language (L2), specifically regarding the linguistic (orthographic, semantic, syntactic, lexical) and cognitive (verbal reasoning, lexical decision) components for the immigrant population in Portugal, a study of cut-off points of 6 tests was conducted. This study was motivated by the unknown tests and the gap concerning the testing and rating of individuals in L2 settings, different from English as L2. The objective was to validate and inform about new tests (and how to establish rating scores and understand the proficiency level of students) in Second Language Testing (for European Portuguese) especially concerning different psychometrics for specific skills (e.g., phonemic blending). In this way, we contribute to identify new procedures for schools and professionals about testing of cognitive functions in immigrant populations. First, we examined 108 non-native students attending Portuguese schools, aged 7–17 years, speakers of mother tongues other than Portuguese and residing in Portugal for a period not exceeding eight years. Then, series of univariate analyses, non-parametric tests and the calculation of percentiles enabled the subsequent classification of the subjects’ proficiency levels: With or without proficiency. Several levels were identified. The sensitivity and specificity indexes were calculated using the Receiver Operating Characteristic (ROC) analysis to define the cut-off points for each test. The score calculation took into account the chronological age, age of immigration and gender variables. The evidence from this study suggests that all tasks are suitable to assess immigrant students. However with the ROC analysis, this assessment differs regarding degrees of proficiency between groups given the characteristics of the tasks and of the subjects. The importance of doing tests with discriminatory power of the subjects’ performance to be able to intervene in the linguistic and cognitive areas with the greatest deficit is examined here.