Large-scale Educational Assessment Research Articles

No substantive effects of school socioeconomic composition on student achievement in Australia: a response to Sciffer, Perry and McConney

In this journal, Sciffer et al. (Large-scale Assessments in Education 10:1–22, 2022), hereafter SP&M, conclude that school socioeconomic compositional (SEC) or school socioeconomic status (school-SES) effects in Australia are substantial and substantively important for research and policy. This paper demonstrates that these claims are unwarranted. Their SEC estimates are much larger than estimates from comparable studies and a metastudy. Despite plausible theoretical reasons and empirical evidence, SP&M do not consider that school academic composition is a significant predictor of student achievement independent of SEC. SEC effects are confounded by academic composition and are typically trivial when considering academic composition. The second part of this paper compares SP&M’s estimates with analysis of the same data, from the Australian National Assessments in Performance—Literacy and Numeracy (NAPLAN). In a model corresponding to SP&M analyses comprising demographics, SES, school-SES, and student-level prior achievement, the effects of school-SES are small, with standardized effects mostly less than 0.10. With the addition of academic composition measured by school-level prior achievement, school-SES effects are effectively zero. In contrast, academic composition has significant, albeit small, impacts on student achievement. Therefore, contrary to SP&M’s (2022) conclusion, school-SES effects on student achievement in NAPLAN are negligible, whereas school-level prior achievement has small effects. That is not to say that school-SES is always irrelevant, but any assessment of its importance must consider both student- and school-level prior achievement.

Evaluating the Douglas-Cohen IRT Goodness of Fit Measure With BIB Sampling of Items.

Methods to detect item response theory (IRT) item-level misfit are typically derived assuming fixed test forms. However, IRT is also employed with more complicated test designs, such as the balanced incomplete block (BIB) design used in large-scale educational assessments. This study investigates two modifications of Douglas and Cohen's 2001 nonparametric method of assessing item misfit, based on A) using block total score and B) pooling booklet level scores for analyzing BIB data. Block-level scores showed extreme inflation of Type I error for short blocks containing 5 or 10 items. The pooled booklet method yielded Type I error rates close to nominal in most conditions and had power to detect misfitting items. The study also found that the Douglas and Cohen procedure is only slightly affected by the presence of other misfitting items in the block. The pooled booklet method is recommended for practical applications of Douglas and Cohen's method with BIB data.

Using plausible values when fitting multilevel models with large-scale assessment data using R

The use of large-scale assessments (LSAs) in education has grown in the past decade though analysis of LSAs using multilevel models (MLMs) using R has been limited. A reason for its limited use may be due to the complexity of incorporating both plausible values and weighted analyses in the multilevel analyses of LSA data. We provide additional functions in R that extend the functionality of the WeMix (Bailey et al., 2023) package to allow for the automatic pooling of plausible values. In addition, functions for model comparisons using plausible values and the ability to export output to different formats (e.g., Word, html) are also provided.

Comparing Examinee-Based and Response-Based Motivation Filtering Methods in Remote Low-Stakes Testing

ABSTRACT Large-scale educational assessments are sometimes considered low-stakes, increasing the possibility of confounding true performance level with low motivation. These concerns are amplified in remote testing conditions. To remove the effects of low effort levels in responses observed in remote low-stakes testing, several motivation filtering methods can be used to purify the data. We estimated scores from assessment data collected remotely in Spring 2021 six ways, applying examinee-based filtering methods (filtering examinees based on total time) and response-based filtering methods (filtering responses using the effort-moderated IRT model), varying the thresholds selected to separate effortful and non-effortful responses. We compared the 2021 scores (estimated six ways) to those obtained in previous cohorts tested in person. The results support the use of motivation filtering regardless of which method is used. Practical implications regarding method and threshold selection are discussed.

Exploring the Efficacy and Implementation of an Online Examination System: A Comprehensive Study on eXamPro in Modern Education

Abstract: Online assessments play a crucial role in evaluating the academic capabilities of students, particularly those enrolling in computer courses or technology-related programs. This model outlines an online placement assessment system, considering the perspectives of the examination process, end-users, and software development. The online assessment system is a digital platform where evaluations are conducted over the internet or intranet using computer systems. It presents an effective solution for large-scale educational assessments. This online educational system incorporates features such as test processing and electronic record-keeping. In this framework, instructors create course-specific questions, forming the basis for online assignments. Users engaging with the platform can access the electronic information they provided and utilize various functions within the online educational system to participate in online assessments. Participants have the opportunity to take multimedia enhanced quizzes related to their courses and submit electronic responses. Upon completing the test duration, users receive grades or marks secured in their assessments. The system performs assessments and auto-grading for multiple-choice questions, streamlining the evaluation process. This online assessment system not only facilitates efficient evaluation but also provides a dynamic and interactive educational experience for both instructors and students in the digital learning environment.

On the Performance of Horseshoe Priors for Inducing Sparsity in Structural Equation Models

The present work focuses on the performance of two types of shrinkage priors—the horseshoe prior and the recently developed regularized horseshoe prior—in the context of inducing sparsity in path analysis and growth curve models. Prior research has shown that these horseshoe priors induce sparsity by at least as much as the “gold standard” spike-and-slab prior. The horseshoe priors are compared to the ridge prior and lasso prior, as well as default non-informative priors, in terms of the percent shrinkage in the model parameters and out-of-sample predictive performance. Empirical studies using data from two large-scale educational assessments reveal the clear advantages of the horseshoe priors in terms of both shrinkage and predictive performance. Simulation studies reveal clear advantages in terms of shrinkage, but less obvious advantages in terms of predictive performance, except in the small sample size condition where both horseshoe priors provide noticeably improved predictive performance.

Challenges for the Design of International Assessments: Sampling, Measurement, and Causality

This paper examines critical design factors that influence data quality in educational research, using international large-scale educational assessments as an example. We will focus on statistical challenges related to sampling, measurement, and causality. While international assessments employ rigorous random sampling techniques, deviations such as exclusions and non-participation can introduce bias and affect representativeness. In terms of measurement, although these assessments excel in core domains, there is a growing call for broader assessment areas, such as environmental literacy and civic education. Additionally, concerns are emerging about the quality of context surveys. Causality remains a central concern, and despite the challenges posed by the cross-sectional design, combining data and applying sophisticated analytical methods can help address causal questions. Recognising the interconnectedness of sampling, measurement, and causality is essential for conducting robust research and informing evidence-based policies and practices.

Cybersecurity: a tool for curbing examination breaches and improvement of the quality of large-scale educational assessments

ABSTRACT This study investigated how cybersecurity can be integrated to curb examination breaches among large-scale educational assessment bodies in Nigeria. Descriptive research designed guided the study with 1067 participants drawn through multistage sampling procedure. Cybersecurity and examination breaches on large-scale educational assessment questionnaire (CEBLSEA-Q) was used for data collection. The internal consistency reliability index of the instrument was evaluated to be .78 using Cronbach alpha method. The findings of the study showed that hacking of websites, colluding via social media (WhatsApp, Facebook, Twitter etc.), sending of malicious/phishing links, using man-in-the-middle (MITM) attack are some of the cyberattacks on large-scale educational assessments. The proportion of examination breaches experienced by the various examination bodies was significantly different with WAEC being the highest, followed by NECO, JAMB, and NABTEB. Cybersecurity measures such as cloud computing, securing domain name server (DNS), encrypting computers and applications containing assessment database, cybersecurity awareness, among others could curb cyberspace aided examination breaches on educational assessments. Cybersecurity measures should be adopted by large-scale educational agencies in order to curb examination breaches that are aided through the cyberspace.

Measuring education: Do we need a plethora of assessment studies or just a single score?

The major international large-scale assessments in education prioritize the three core domains such as mathematics, science, and reading, neglecting other important domains such as civic education or computer and information literacy. The present study aims at determining the added value of different international large-scale assessments to evaluating educational systems across the globe. Using country-level data from all recent international assessments by the IEA and the OECD, we conducted a series of correlational analyses and multidimensional scaling to determine the similarities and differences among educational assessments. The findings of this study show that the various measures of achievement correlate to different degrees at country level. While mathematics, science, and reading are the most closely correlated, the correlations with civic education and computer and information literacy are considerably lower. We conclude that subject domains as civic education and computer and information literacy add a significant value to the assessment of education.

Explaining performance decline over the course of taking comprehensive proficiency tests: the roles of effort and omission propensity

In order to ensure content validity by covering a broad range of content domains, the testing times of some educational large-scale assessments last up to a total of two hours or more. Performance decline over the course of taking the test has been extensively documented in the literature. It can occur due to increases in the numbers of (a) incorrect responses or (b) missing responses. Both reasons can be regarded as the result of decreasing effort. We investigated the roles of self-reported effort and omission propensity for performance decline over the course of taking a comprehensive educational large-scale assessment in Germany. Latent change score models showed that the effects of initial effort and change in effort on decline in performance remained almost unchanged when we added omission propensity and change in omission propensity as predictors of performance decline. This indicates that self-reported effort and omission propensity capture different aspects of performance decline. This finding held in both proficiency domains we investigated: math and science. When developing strategies for mitigating threats to test score validity, the two constructs (i.e., effort and omission propensity) may be considered separately.

Detecting Differential Item Functioning in 2PL Multistage Assessments

The detection of differential item functioning is crucial for the psychometric evaluation of multistage tests. This paper discusses five approaches presented in the literature: logistic regression, SIBTEST, analytical score-based tests, bootstrap score-based tests, and permutation score-based tests. First, using an simulation study inspired by a real-life large-scale educational assessment, we compare the five approaches with respect to their type I error rate and their statistical power. Then, we present an application to an empirical data set. We find that all approaches show type I error rates close to the nominal alpha level. Furthermore, all approaches are shown to be sensitive to uniform and non-uniform DIF effects, with the score-based tests showing the highest power.

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

From asking to observing. Behavioural measures of socio-emotional and motivational skills in large-scale assessments

Socio-emotional and motivational skills are routinely measured using self-reports in large-scale educational assessments. Measures exploiting test-takers’ behaviour during the completion of questionnaires or cognitive tests are increasingly used as alternatives to self-reports in the economics of education literature. We compute behavioural measures of socio-emotional and motivational skills using data from the Programme for International Student Assessment (PISA). We find that these measures capture important aspects of students' academic profiles: some are importantly associated with contemporaneous performance and educational attainment and most measures have a high degree of stability over time. However, these measures are only limitedly correlated among themselves and have low correlations with self-report measures of the same constructs. This is likely a reflection of the fact that behavioural measures are representations of the test taker current ‘state’, rather than descriptions of the participant view of their own ‘trait’ like the self-report measures. Moreover, the low correlation across measures suggests that they capture different behavioural responses to the test-taking situation. These differences are still limitedly understood because the measures are constructed ex-post using collateral information collected during the administration of assessments rather than developed ex ante in line with theoretical models of human cognition and affect.

Fast estimation of multiple group generalized linear latent variable models for categorical observed variables

A computationally efficient method for marginal maximum likelihood estimation of multiple group generalized linear latent variable models for categorical data is introduced. The approach utilizes second-order Laplace approximations of the integrals in the likelihood function. It is demonstrated how second-order Laplace approximations can be utilized highly efficiently for generalized linear latent variable models by considering symmetries that exist for many types of model structures. In a simulation with binary observed variables and four correlated latent variables in four groups, the method has similar bias and mean squared error compared to adaptive Gauss-Hermite quadrature with five quadrature points while substantially improving computational efficiency. An empirical example from a large-scale educational assessment illustrates the accuracy and computational efficiency of the method when compared against adaptive Gauss-Hermite quadrature with three, five, and 13 quadrature points.

Machine learning procedures for predictor variable selection for schoolwork-related anxiety: evidence from PISA 2015 mathematics, reading, and science assessments

BackgroundLarge-scale international studies offer researchers a rich source of data to examine the relationship among variables. Machine learning embodies a range of flexible statistical procedures to identify key indicators of a response variable among a collection of hundreds or even thousands of potential predictor variables. Among these, penalized regression approaches, including least absolute selection and shrinkage operator (LASSO) and elastic net (Enet), have been advanced as useful tools capable of handling large number of predictors for variable selection for model generation. While the utility of penalized regression within educational research is emerging, less application of these machine learning methods, including random forest, to predictor variable selection in large-scale international data appears in the literature. In response, this study compared LASSO, Enet, and random forest for predictor variable selection, including the traditional forward stepwise (FS) regression approach, for students’ test anxiety or, more specifically, schoolwork-related anxiety based on PISA 2015 data.MethodsPrediction of the three machine learning methods were compared for variable selection of 188 indicators of schoolwork-related anxiety. Data were based on US students (N = 5593) who participated in PISA 2015. With the exception of FS, LASSO, Enet, and random forest were iterated 100 times to consider the bias resulting from data-splitting to determine the selection or non-selection of each predictor. This resulted in the reporting of number of selected variables into the following five count categories: 1 or more, 25 or more, 50 or more, 75 or more, and all 100 iterations.ResultsLASSO and Enet both outperformed random forest but did not differ from one another in terms of prediction performance in 100 iterations of modeling. Correspondingly, LASSO was compared to FS in which, of the 188 predictors, 27 were identified as key indicators of schoolwork-related anxiety across 100 iterations, and 26 variables were also statistically significant with FS regression. Aligned with previous research, key indicators included personal, situational, and mathematics and reading achievement. Further, LASSO identified 28 variables (14.89%) statistically unrelated to schoolwork-related anxiety, which included indicators aligned to students’ academic- and non-academic behaviors.ConclusionsLASSO and Enet outperformed random forest and yielded comparable results in which determinants of schoolwork-related anxiety included personal and environmental factors, including achievement goals, sense of belonging, and confidence to explain scientific phenomenon. LASSO and FS also identified similar predictor variables related, as well as unrelated, to schoolwork-related anxiety. Aligned with previous research, females reported higher schoolwork-related anxiety than males. Mathematics achievement was negatively related to anxiety, whereas reading performance was positively associated with anxiety. This study also bears significance as one of the first penalized regression studies to incorporate sampling weights and reflect the complex sampling schemes of large-scale educational assessment data.

NEPSscaling: plausible value estimation for competence tests administered in the German National Educational Panel Study

Educational large-scale assessments (LSAs) often provide plausible values for the administered competence tests to facilitate the estimation of population effects. This requires the specification of a background model that is appropriate for the specific research question. Because the German National Educational Panel Study (NEPS) is an ongoing longitudinal LSA, the range of potential research questions and, thus, the number of potential background variables for the plausible value estimation grow with each new assessment wave. To facilitate the estimation of plausible values for data users of the NEPS, the R package NEPSscaling allows their estimation following the scaling standards in the NEPS without requiring in-depth psychometric expertise in item response theory. The package requires the user to prepare the data for the background model only. Then, the appropriate item response model including the linking approach adopted for the NEPS is selected automatically, while a nested multiple imputation scheme based on the chained equation approach handles missing values in the background data. For novice users, a graphical interface is provided that only requires minimal knowledge of the R language. Thus, NEPSscaling can be used to estimate cross-sectional and longitudinally linked plausible values for all major competence assessments in the NEPS.

IQB Trends in Student Achievement 2018: A Large-Scale Educational Assessment Study in Germany

The <em>IQB Trends in Student Achievement 2018</em> is a nation-wide large-scale educational assessment using a trend design. The study includes achievement tests on mathematics, biology, chemistry, and physics as well as background questionnaires for students, their parents, subject teachers, and school principals. The datasets include data of 51.511 grade-9 students from 1.473 German schools assessed in 2018, data of 5.026 teachers, and 1.264 school principals. The 6 datasets and the study documentation are available via the Research Data Centre (FDZ) at IQB and include a wide variety of constructs such as achievement estimates, indicators of general cognitive abilities, motivational and socio-emotional constructs as well as data on social networks.

Validating school-based measures of educational disadvantage in Ireland

Reducing the socio-economic achievement gap is a key goal of education policy internationally. Since 2017, the stated intention in Ireland is to use the Pobal HP Deprivation Index to identify schools serving high proportions of disadvantaged students. In spite of this, little published research has compared the performance of the HP index to richer survey measures from large-scale educational assessments or examined its association with educational achievement at primary or second level. This paper aims to assess the validity and fitness for purpose of the HP index for use in identifying schools serving high concentrations of socio-economically disadvantaged students. Analyses draw on Ireland’s Program for International Student Assessment (PISA) 2018 dataset matched with administrative data. Findings show a strong correlation between school-average HP and other socio-economic measures examined. However, about one-in-five schools are identified as potential ‘false positives’ or ‘false negatives’ when school-average HP is used for identification purposes. Also, school-average HP explains less variance in reading achievement than other variables considered. Conclusions recognize the benefits of the HP index but emphasize the need for ongoing examination of the most appropriate methods of identification.

Identifying the Predictors of Mathematics Anxiety AND Performance in Canada: An Educational Data Mining Approach

Over the last decade, Canadian students have exhibited insubstantial improvements in mathematical scores compared to other countries as indicated by large-scale educational assessments such as the Programme for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS). In relation to students’ mathematical performance, math anxiety - the feeling of fear or nervousness when performing math-related tasks - was found as an associated factor. However, no previous study has explored math performance and math anxiety, specifically among Albertan students. We present a work-in-progress that identifies significant predictors of math performance and math anxiety among Canadian and Albertan students, using the PISA 2018 and TIMSS 2019 datasets. This study has three phases: first, a list of predictors will be selected from the data set based on existing theories regarding students’ math performance and math anxiety. The initial list of predictors will be presented to domain experts (i.e., math teachers) for refinement based on their practical experience. A predictive model for math performance and math anxiety will be developed with Educational Data Mining techniques in the second phase. Results from the model will be presented to the domain experts for their inputs as the qualitative component, and variable importance metrics of the model will be consulted for the quantitative component. Findings from both components will be integrated consulted with the domain experts to derive actionable recommendations that would inform various stakeholders (e.g., educators, school districts, and Alberta Education) of ways to improve math performance in Alberta students.

Power Analysis for the Wald, LR, Score, and Gradient Tests in a Marginal Maximum Likelihood Framework: Applications in IRT

The Wald, likelihood ratio, score, and the recently proposed gradient statistics can be used to assess a broad range of hypotheses in item response theory models, for instance, to check the overall model fit or to detect differential item functioning. We introduce new methods for power analysis and sample size planning that can be applied when marginal maximum likelihood estimation is used. This allows the application to a variety of IRT models, which are commonly used in practice, e.g., in large-scale educational assessments. An analytical method utilizes the asymptotic distributions of the statistics under alternative hypotheses. We also provide a sampling-based approach for applications where the analytical approach is computationally infeasible. This can be the case with 20 or more items, since the computational load increases exponentially with the number of items. We performed extensive simulation studies in three practically relevant settings, i.e., testing a Rasch model against a 2PL model, testing for differential item functioning, and testing a partial credit model against a generalized partial credit model. The observed distributions of the test statistics and the power of the tests agreed well with the predictions by the proposed methods in sufficiently large samples. We provide an openly accessible R package that implements the methods for user-supplied hypotheses.