Large-scale Educational Assessment Research Articles

Automatically analyzing text responses for exploring gender-specific cognitions in PISA reading

BackgroundThe gender gap in reading literacy is repeatedly found in large-scale assessments. This study compared girls’ and boys’ text responses in a reading test applying natural language processing. For this, a theoretical framework was compiled that allows mapping of response features to the preceding cognitive components such as micro- and macropropositions from the situation model.MethodsIn total, n = 33,604 responses from the German sample of the Programme for International Student Assessment (PISA) 2012 reading test have been analyzed for characterizing the genders’ typical cognitive approaches. The analyses mainly explored the gender gap by contrasting groups of responses typical for either gender. These gender-specific responses characterize the typical responding of the genders to PISA reading questions.ResultsResponses typical for girls contained three to five more proposition entities from the situation model, irrespective of the response correctness. They integrated more relevant propositions and constituted better fits to the question focus. That means, in answering questions which ask for explicit information from the stimulus text, the typical girl responses appropriately encompassed more micropropositions, and typical boy responses tended to include more macropropositions—vice versa for questions requesting implicit information.ConclusionIt appears that typical boy responses to PISA reading questions are characterized by struggling with retrieving and integrating propositions from the situation model. The typical girl liberally juggles these to formulate the responses. The results demonstrate that text responses are a neglected but informative source for educational large-scale assessments made accessible through natural language processing.

Reliability and Validity of PIRLS and TIMSS

Abstract. Academic achievements are often assessed in written exams and tests using selection-type (e.g., multiple-choice, MC) and supply-type (e.g., constructed-response, CR) item response formats. The present article examines how MC items and CR items differ with regard to reliability and criterion validity in two educational large-scale assessments with 4th-graders. The reading items of PIRLS 2006 were compiled into MC scales, CR scales, and mixed scales. Scale reliabilities were estimated according to item response theory (international PIRLS sample; n = 119,413). MC showed smaller standard errors than CR around the reading proficiency mean, whereas CR was more reliable for low and high proficiency levels. In the German sample (n = 7,581), there was no format-specific differential validity (criterion: German grades, r ≈ .5; Δr = 0.01). The mathematics items of TIMSS 2007 (n = 160,922) showed similar reliability patterns. MC validity was slightly larger than CR validity (criterion: mathematics grades; n = 5,111; r ≈ .5, Δr = −0.02). Effects of format-specific test extensions were very small in both studies. It seems that in PIRLS and TIMSS, reliability and validity do not depend substantially on response formats. Consequently, other response format characteristics (like the cost of development, administration, and scoring) should be considered when choosing between MC and CR.

On imputation for planned missing data in context questionnaires using plausible values: a comparison of three designs

BackgroundThis paper extends a recent study by Kaplan and Su (J Educ Behav Stat 41: 51–80, 2016) examining the problem of matrix sampling of context questionnaire scales with respect to the generation of plausible values of cognitive outcomes in large-scale assessments.MethodsFollowing Weirich et al. (Nested multiple imputation in large-scale assessments. In: Large-scale assessments in education, 2. http://www.largescaleassessmentsineducation.com/content/2/1/9, 2014) we examine single + multiple imputation and multiple + multiple imputation methods using predictive mean matching imputation under three different context questionnaire matrix sampling designs: a two-form design studied by Adams et al. (On the use of rotated context questionnaires in conjunction with multilevel item response models. In: Large-scale assessments in education. http://www.largescaleassessmentsineducation.com/content/1/1/5, 2013), a three-form design implemented in PISA 2012, and a partially-balanced incomplete design studied by Kaplan and Su (J Educ Behav Stat 41: 51–80, 2016).ResultsOur results show that the choice of design has a larger impact on the reduction of bias than the choice of imputation method. Specifically, the three-form design used in PISA 2012 yields considerably less bias compared to the two-form design and the partially balanced incomplete design. We further show that the partially balanced incomplete block design produces less bias than the two-form design despite having the same amount of missing data.ConclusionsWe discuss the results in terms of implications for the design of context questionnaires in large-scale assessments.

Optimizing Prediction Using Bayesian Model Averaging: Examples Using Large-Scale Educational Assessments.

This article provides a review of Bayesian model averaging as a means of optimizing the predictive performance of common statistical models applied to large-scale educational assessments. The Bayesian framework recognizes that in addition to parameter uncertainty, there is uncertainty in the choice of models themselves. A Bayesian approach to addressing the problem of model uncertainty is the method of Bayesian model averaging. Bayesian model averaging searches the space of possible models for a set of submodels that satisfy certain scientific principles and then averages the coefficients across these submodels weighted by each model's posterior model probability (PMP). Using the weighted coefficients for prediction has been shown to yield optimal predictive performance according to certain scoring rules. We demonstrate the utility of Bayesian model averaging for prediction in education research with three examples: Bayesian regression analysis, Bayesian logistic regression, and a recently developed approach for Bayesian structural equation modeling. In each case, the model-averaged estimates are shown to yield better prediction of the outcome of interest than any submodel based on predictive coverage and the log-score rule. Implications for the design of large-scale assessments when the goal is optimal prediction in a policy context are discussed.

18REST: a short RIASEC-interest measure for large-scale educational and vocational assessment

The construction of the 18REST, a short 18-item inventory to describe students’ position on John Holland’s RIASEC interest types, is documented. The instrument is meant to be used in large-scale assessment in education and on the labor market, supplementing information on school achievement and social-emotional skills. This research was carried out in Brazil, initially with two independent samples composed by adolescents and adults. The 18REST’s psychometric properties are compared to those of the more extended RIASEC item pool and confirmed in a new independent undergraduate sample. Despite differences between genders were found as expected, invariance measurement across gender was indicated. Different ways to use the 18REST in large-scale assessment are discussed.

League tables in educational evidence-based policy-making: can we stop the horse race, please?

ABSTRACTInternational large-scale student assessments (ILSAs) in education represent a valuable source of information for policy-makers, not only on student achievements, but also on their relationship with different contextual factors. The results are partly described in the official studies’ reports; more can be derived from the publicly released data sets. However, league tables are often the only evidence used in policy debates and decisions on education. Indeed, the comparison of student achievement across the participating educational systems is a legitimate proxy for estimating countries’ development and productivity, but the use of league tables more often turns into ‘horse-ranking’, ignoring the contexts of teaching and learning. This is often supported by the media, turning the use of results into their abuse. The purpose of this paper is to discuss the use and misuse of league tables in reporting ILSA results, vs. the use of data for in-depth analysis in order to make informed decisions.

Using Procedure Based on Item Response Theory to Evaluate Classification Consistency Indices in the Practice of Large-Scale Assessment.

In spite of the growing interest in the methods of evaluating the classification consistency (CC) indices, only few researches are available in the field of applying these methods in the practice of large-scale educational assessment. In addition, only few studies considered the influence of practical factors, for example, the examinee ability distribution, the cut score location and the score scale, on the performance of CC indices. Using the newly developed Lee's procedure based on the item response theory (IRT), the main purpose of this study is to investigate the performance of CC indices when practical factors are taken into consideration. A simulation study and an empirical study were conducted under comprehensive conditions. Results suggested that with negatively skewed distribution, the CC indices were larger than with other distributions. Interactions occurred among ability distribution, cut score location, and score scale. Consequently, Lee's IRT procedure is reliable to be used in the field of large-scale educational assessment, and when reporting the indices, it should be treated with caution as testing conditions may vary a lot.

Assessment and learning: fields apart?

Educational assessments define what aspects of learning will formally be given credit and therefore have a huge impact upon teaching and learning. Although the impact of high-stakes national and international assessments on teaching and learning is considered in the literature, remarkably, there is little research on the connection between theories of learning and educational assessments. Given the voluminous assessment that takes place annually in systematic ways in most many nations, it is surprising that more has not been gained from these assessments in the development of theories of learning and vice versa. In this article, we consider both theories of learning and assessment and draw the main message of the article, that if assessments are to serve the goals of education, then theories of learning and assessment should be developing more closely with each other. We consider fundamental aspects of assessment theory, such as constructs, unidimensionality, invariance and quantifiability, and in doing so, we distinguish between educational and psychological assessment. Second, we show how less traditionally considered cases of (a) international assessments and (b) Assessment for Learning affect student learning. Through these cases we illustrate the otherwise somewhat theoretical discussion in the article. We argue that if assessment is to serve the learning goals of education, then this discussion on the relationship between assessment and learning should be developed further and be at the forefront of high-stakes, large-scale educational assessments.

Evaluating the risk of nonresponse bias in educational large-scale assessments with school nonresponse questionnaires: a theoretical study

Survey participation rates can have a direct impact on the validity of the data collected since nonresponse always holds the risk of bias. Therefore, the International Association for the Evaluation of Educational Achievement (IEA) has set very high standards for minimum survey participation rates. Nonresponse in IEA studies varies between studies and cycles. School participation is at a higher risk relative to within-school participation; school students are more likely to cooperate than adults (i.e., university students or school teachers). Across all studies conducted by the IEA during the last decade, between 7 and 33% of participating countries failed to meet the minimum participation rates at the school level. Quantifying the bias introduced by nonresponse is practically impossible with the currently implemented design. During the last decade social researchers have introduced and developed the concept of nonresponse questionnaires. These are shortened instruments applied to nonrespondents, and aim to capture information that correlates with both: survey’s main outcome variable(s), and respondent’s propensity of participation. We suggest in this paper a method to develop such questionnaires for nonresponding schools in IEA studies. By these means, we investigated school characteristics that are associated with students’ average achievement scores using correlational and multivariate regression analysis in three recent IEA studies. We developed regression models that explain with only 11 school questionnaire variables or less up to 77% of the variance of the school mean achievement score. On average across all countries, the R2 of these models was 0.24 (PIRLS), 0.34 (TIMSS, grade 4) and 0.36 (TIMSS grade 8), using 6–11 variables. We suggest that data from such questionnaires can help to evaluate bias risks in an effective way. Further, we argue that for countries with low participation rates a change in the approach of computing nonresponse adjustment factors to a system were school´s participation propensity determines the nonresponse adjustment factor should be considered.

Projecting images of the ‘good’ and the ‘bad school’: top scorers in educational large-scale assessments as reference societies

Researchers interested in the global flow of educational ideas and programmes have long been interested in the role of so-called ‘reference societies’. The article investigates how top scorers in large-scale assessments are framed as positive or negative reference societies in the education policy-making debate in German mass media and which functions they fulfil. Top scores in large-scale assessments do not automatically promote a country to the status of a positive reference society. Whether top scorers are perceived as positive or as negative reference societies depends largely on stereotyped prior perceptions that determine how success in these assessments is framed. Among the functions positive and negative reference societies fulfil are making educational reform agendas more plausible and serving as projection screens for conceptions of the ‘good’ and the ‘bad’ school.

Policy Implications of Achievement Testing Using Multilevel Models: The Case of Brazilian Elementary Schools.

Large-scale educational assessment has been established as source of descriptive, evaluative and interpretative information that influence educational policies worldwide throughout the last third of the twentieth century. In the 1990s the Brazilian Ministry of Education developed the National Basic Education Assessment System (SAEB) that regularly measures management, resource and contextual school features and academic achievement in public and private institutions. In 2005, after significant piloting and review of the SAEB, a new sampling strategy was taken and Prova Brasil became the new instrument used by the Ministry to assess skills in Portuguese (reading comprehension) and Mathematics (problem solving), as well as collecting contextual information concerning the school, principal, teacher, and the students. This study aims to identify which variables are predictors of academic achievement of fifth grade students on Prova Brasil. Across a large sample of students, multilevel models tested a large number of variables relevant to student achievement. This approach uncovered critical variables not commonly seen as significant in light of other achievement determinants, including student habits, teacher ethnicity, and school technological resources. As such, this approach demonstrates the value of MLM to appropriately nuanced educational policies that reflect critical influences on student achievement. Its implications for wider application for psychology studies that may have relevant impacts for policy are also discussed.

IRT Item Parameter Scaling for Developing New Item Pools

ABSTRACTIncreasing use of item pools in large-scale educational assessments calls for an appropriate scaling procedure to achieve a common metric among field-tested items. The present study examines scaling procedures for developing a new item pool under a spiraled block linking design. The three scaling procedures are considered: (a) concurrent calibration, (b) separate calibration with one linking, and (c) separate calibration with three sequential linking. Evaluation across varying sample sizes and item pool sizes suggests that calibrating an item pool simultaneously results in the most stable scaling. The separate calibration with linking procedures produced larger scaling errors as the number of linking steps increased. The Haebara’s item characteristic curve linking resulted in better performances than the test characteristic curve (TCC) linking method. The present article provides an analytic illustration that the test characteristic curve method may fail to find global solutions in polytomous items. Finally, comparison of the single- and mixed-format item pools suggests that the use of polytomous items as the anchor can improve the overall scaling accuracy of the item pools.

Current issues in large-scale educational assessment in Japan: focus on national assessment of academic ability and university entrance examinations

Currently, large-scale testing in Japan faces conflicting requirements derived from principles of education on the one hand, and measurement, on the other. Issues of affective ambivalence towards tests (i.e. test aversion and dependence) are also observed. The seemingly conflicting government discussions regarding the national assessment of academic achievement at primary and middle schools, and reforms to university entrance examinations, are discussed here in terms of these issues.

Assessment of fit of item response theory models used in large-scale educational survey assessments

Latent regression models are used for score-reporting purposes in large-scale educational survey assessments such as the National Assessment of Educational Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS). One component of these models is based on item response theory. While there exists some research on assessment of fit of item response theory models in the context of large-scale assessments, there is a scope of further research on the topic. We suggest two types of residuals to assess the fit of item response theory models in the context of large-scale assessments. The Type I error rates and power of the residuals are computed from simulated data. The residuals are computed using data from four NAEP assessments. Misfit was found for all data sets for both types of residuals, but the practical significance of the misfit was minimal.

Causal inference with large‑scale assessments in education from a Bayesian perspective: a review and synthesis.

This paper reviews recent research on causal inference with large-scale assessments in education from a Bayesian perspective. I begin by adopting the potential outcomes model of Rubin (J Educ Psychol 66:688–701, 1974) as a framework for causal inference that I argue is appropriate with large-scale educational assessments. I then discuss the elements of Bayesian inference arguing that methods and models of causal inference can benefit from the Bayesian approach to quantifying uncertainty. Next I outline one method of causal inference that I believe is fruitful for addressing causal questions with large-scale educational assessments within the potential outcomes framework— namely, propensity score analysis. I then discuss the quantification of uncertainty in propensity score analysis through a Bayesian approach. Next, I discuss a series of necessary conditions for addressing causal questions with large-scale educational assessments. The paper closes with a discussion of the implications for the design of large-scale educational assessments when the goal is in asking causal questions and warranting causal claims.

La revolución digital y la evaluación: un nuevo paradigma

La revolución digital ha dado lugar a una nueva era en la evaluación educativa a gran escala; ofrece posibilidades que hace una década eran impensables. Junto con los avances de las ciencias cognitivas y la psicometría fue posible lograr avances sustanciales, como la generación automática de reactivos. El propósito de este trabajo es presentar un nuevo paradigma de evaluación, basado en campos semánticos, que se vale de la tecnología digital. Este paradigma permite evaluar competencias cognitivas de mayor complejidad, a través de respuestas construidas o semi-construidas y de crédito parcial, superando así muchas de las limitaciones del formato tradicional de lápiz y papel y de opción múltiple. Se describen las características teóricas, conceptuales, técnicas y computacionales del Examen de Competencias Básicas (EXCOBA) y el Generador Automático de Exámenes (GenerEx), lo que permite generar ítems isomorfos y exámenes equivalentes.

Data imputation algorithms for mixed variable types in large scale educational assessment: a comparison of random forest, multivariate imputation using chained equations, and MICE with recursive partitioning

Missing data is a major issue with which researchers working on large scale assessments must contend. Such research efforts frequently collect a wide array of variables, including dichotomous, ordinal, nominal, normal, skewed, and counts. This variation in data distributions renders many recommended methods for missing data imputation less than optimal because they assume a single joint probability model for all variables. This simulation study compared four imputation methods, random forest imputation (RF), multivariate imputation by chained equations (MICE), and combinations of the two methods using either the recursive partitioning tree (MICE-RPT) or random forest (MICE-RF) methodologies. Results reveal that data imputed with RF, MICE, MICE-RF, and MICE-RPT yield more accurate parameter estimates than data treated with LD and that MICE-RF and MICE-RPT are associated with more accurate estimates than MICE or RF alone. Implications of these results and recommendations for practice are discussed.

Measurement Issues in Large-Scale Educational Assessment

Measurement Issues in Large-Scale Educational Assessment

Understanding civic cognitive assessment tasks: Associations between linguistic features and students’ task performance

Abstract This study investigates how text features and item characteristics are associated with early adolescents’ performance in large-scale assessments in civic education (CIVED). Examining responses to multiple-choice items from the IEA CIVED assessment, the study identified linguistic features related to item difficulty. These features account for more than one-third of the variance in difficulty level of 38 test items for a national sample of 3000 US ninth-grade students. On this test in CIVED short words, words with concrete meanings, intentional verbs and certain noun phrases were likely to facilitate students’ comprehension and ability to correctly answer an item. Contrariwise, lengthy words, words with abstract meanings and complex syntactic structure hindered student performance. Some linguistic features significantly predict the performance of students from homes with high literacy resources, but do not reach significance for less advantaged students. Kintsch’s situation model helps to interpret these results.

Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving

This paper aims at showcasing the potential of log-file analyses by capitalizing on the computer-based assessment of complex problem solving (CPS) in the 2012 cycle of the Programme for International Student Assessment (PISA). We analyzed log-file data from the CPS item Climate Control encompassing N = 16,219 students from 44 countries and economies. In Research Question 1, we related the application of an optimal exploration strategy (i.e., vary-one-thing-at-at-time, VOTAT) to performance on this specific problem and to overall level of CPS proficiency, both on an individual and on the country level. In Research Question 2, we identified several groups of students with different levels of non-mastery on a fine-grained level. Results indicated that (1) the VOTAT strategy was strongly related to performance on the Climate Control item as well as to overall problem solving proficiency, and (2) that there were different levels of non-mastery that ranged from applying no systematic strategic behavior to actually applying VOTAT but still failing to solve the item. On the backdrop of these results, we discuss implications and future potentials of log-file analyses in educational large-scale assessments for researchers, teachers, and policy makers.