Examinee Ability Research Articles

Implementing Computer Adaptive Testing for High-Stakes Assessment: A Shift for Examinations Council of Lesotho

We examine the feasibility of implementing Computer Adaptive Testing (CAT) for high-stakes assessments in Lesotho, specifically through the Examination Council of Lesotho (ECoL). CAT, a cutting-edge testing method, enhances precision and efficiency by adjusting test items in real-time based on an examinee's ability. While CAT has gained widespread global adoption, its implementation in developing countries presents significant challenges, particularly regarding infrastructure, expertise, and resource limitations. Our research addresses a critical gap in understanding how CAT can be effectively integrated into educational systems in developing contexts such as Lesotho. We conducted a literature review across five databases—Google Scholar, ERIC, PsycINFO, JSTOR, and PubMed—retrieving 48 studies published between 2013 and 2023. Of these, 18 studies met our inclusion criteria, focusing on CAT's advantages, challenges, and real-world applications in educational assessments. We applied thematic analysis to identify key benefits and barriers, focusing on precision, fairness, and infrastructure requirements. We outline the essential stages of CAT development, including feasibility studies, item bank creation, pretesting and calibration, specification determination, and live CAT publication. We highlight critical tasks such as using Monte Carlo simulations to validate CAT feasibility and developing a robust item bank calibrated with Item Response Theory (IRT). We also address challenges like building a robust technological infrastructure, providing comprehensive stakeholder training, and securing adequate funding. We emphasize the importance of continuous evaluation and stakeholder engagement to ensure CAT's successful implementation and sustainability. Global trends indicate growing adoption, driven by advancements in psychometrics and technology. CAT has the potential to offer more equitable and accurate assessments, making it a promising solution to improve educational outcomes in Lesotho.

An interpretable polytomous cognitive diagnosis framework for predicting examinee performance

As a fundamental task of intelligent education, deep learning-based cognitive diagnostic models (CDMs) have been introduced to effectively model dichotomous testing data. However, it remains a challenge to model the polytomous data within the deep-learning framework. This paper proposed a novel Polytomous Cognitive Diagnosis Framework (PCDF), which employs Cumulative Category Response Function (CCRF) theory to partition and consolidate data, thereby enabling existing cognitive diagnostic models to seamlessly analyze graded response data. By combining the proposed PCDF with IRT, MIRT, NCDM, KaNCD, and ICDM, extensive experiments were complemented by data re-encoding techniques on the four real-world graded scoring datasets, along with baseline methods such as linear-split, one-vs-all, and random. The results suggest that when combined with existing CDMs, PCDF outperforms the baseline models in terms of prediction. Additionally, we showcase the interpretability of examinee ability and item parameters through the utilization of PCDF.

The irtQ R package: a user-friendly tool for item response theorybased test data analysis and calibration.

Computerized adaptive testing (CAT) has become a widely adopted test design for high-stakes licensing and certification exams, particularly in the health professions in the United States, due to its ability to tailor test difficulty in real time, reducing testing time while providing precise ability estimates. A key component of CAT is item response theory (IRT), which facilitates the dynamic selection of items based on examinees' ability levels during a test. Accurate estimation of item and ability parameters is essential for successful CAT implementation, necessitating convenient and reliable software to ensure precise parameter estimation. This paper introduces the irtQ R package (http://CRAN.R-project.org/), which simplifies IRTbased analysis and item calibration under unidimensional IRT models. While it does not directly simulate CAT, it provides essential tools to support CAT development, including parameter estimation using marginal maximum likelihood estimation via the expectation-maximization algorithm, pretest item calibration through fixed item parameter calibration and fixed ability parameter calibration methods, and examinee ability estimation. The package also enables users to compute item and test characteristic curves and information functions necessary for evaluating the psychometric properties of a test. This paper illustrates the key features of the irtQ package through examples using simulated datasets, demonstrating its utility in IRT applications such as test data analysis and ability scoring. By providing a user-friendly environment for IRT analysis, irtQ significantly enhances the capacity for efficient adaptive testing research and operations. Finally, the paper highlights additional core functionalities of irtQ, emphasizing its broader applicability to the development and operation of IRT-based assessments.

Structural validity of the Trunk Assessment Scale for Spinal Cord Injury (TASS) with Rasch analysis for individuals with spinal cord disorders

Objective To confirm the structural validity of the Trunk Assessment Scale for Spinal Cord Injury (TASS). Participants and Methods We evaluated 104 Japanese individuals with a spinal cord injury (SCI) (age 63.5 ± 12.2 years; 64 with tetraplegia) with the TASS 1−3 times. We conducted a Rasch analysis to assess the TASS’ unidimensionality, fit statistics, category probability curve, ceiling/floor effects, local independence, reliability, and difference item function (DIF). Results The TASS was observed to be a unidimensional and highly reproducible scale of item difficulty hierarchy that sufficiently identifies the superiority of the examinee's ability. The TASS was easy for the participants of this study. One TASS item was a misfit based on the infit and outfit mean square; another item also showed a DIF contrast for age. Several items were found to require a synthesis or modification of the content. The TASS showed a floor effect, and most of the non-scorers were individuals with a complete SCI. Conclusion Our findings clarify the structural validity of the TASS, and our analyses revealed that the TASS includes an unfitness item and was less challenging for individuals with SCIs. The improvements suggested by these results provide important information for modifying the TASS to a more useful instrument.

An Illustration of an IRTree Model for Disengagement.

Low-stakes test performance commonly reflects examinee ability and effort. Examinees exhibiting low effort may be identified through rapid guessing behavior throughout an assessment. There has been a plethora of methods proposed to adjust scores once rapid guesses have been identified, but these have been plagued by strong assumptions or the removal of examinees. In this study, we illustrate how an IRTree model can be used to adjust examinee ability for rapid guessing behavior. Our approach is flexible as it does not assume independence between rapid guessing behavior and the trait of interest (e.g., ability) nor does it necessitate the removal of examinees who engage in rapid guessing. In addition, our method uniquely allows for the simultaneous modeling of a disengagement latent trait in addition to the trait of interest. The results indicate the model is quite useful for estimating individual differences among examinees in the disengagement latent trait and in providing more precise measurement of examinee ability relative to models ignoring rapid guesses or accommodating it in different ways. A simulation study reveals that our model results in less biased estimates of the trait of interest for individuals with rapid responses, regardless of sample size and rapid response rate in the sample. We conclude with a discussion of extensions of the model and directions for future research.

Writing Task Performance and First Language Background on an ESL Placement Exam

First Language (L1) has been assumed to play a role in Second Language ability (Bachman & Palmer, 1996). However, the interplay between them across skill, task, or scoring criteria is more complex (Hamp-Lyons & Davies, 2008). Using Many-Facets Rasch Measurement, this study investigates the main effects of examinee ability, rater severity, task difficulty, and rubric scale difficulty and functionality on the writing section of an English as a Second Language program's placement test, then compares performance among L1 Spanish and Japanese examinees to discern the presence of bias across facet. The results for examinee ability and rater severity suggest score variability not expected by the model. Regarding task difficulty and scale difficulty and functionality, it can be concluded that an argumentative essay genre was more difficult than a customer review, or that rater assessed rubric criteria for Content, Organization, and Language more harshly for the former. A bias analysis among L1 Spanish and Japanese examinees revealed that the customer review displayed a bias against Japanese examinees, particularly for Organization, while the argumentative essay displayed bias for them, particularly for Organization and Language. These results demonstrate how placement testing could inform curricula in language programs with linguistically diverse student populations.

Gaining a deeper understanding of the meaning of the carelessness parameter in the 4PL IRT model and strategies for estimating it

Three popular models are used to describe the characteristics of the test items and estimate the ability of examinees under the dichotomous IRT model, namely the one-, two-, and three-parameter logistic models. The three-item parameters are discriminating power, difficulty, and pseudo-guessing. In the development of the dichotomous IRT model, carelessness or upper asymptote parameter was proposed, which forms a four-parameter logistic (4PL) model to accommodate a condition where a high-ability examinee gives an incorrect response to a test item when he/she should be able to respond to the test item correctly. However, the carelessness parameter and the 4PL model have not been widely accepted and used due to several factors, and people’s understanding of that parameter and strategies for estimating it is still inadequate. Therefore, this study aims to shed light on ideas underlying the 4PL model, the meaning of the carelessness parameter, and strategies used to estimate that parameter based on the extant literature. The focus of this study was then extended to demonstrating practical examples of estimating item and person parameters using the 4PL model using empirical data on responses of 1,000 students from the Indonesia Open University (Universitas Terbuka) on 21 of 30 multiple-choice items on the Business English test, a paper-and-pencil test. We mainly analyzed empirical data using the ‘mirt’ package in RStudio. We present the analysis results coherently so that IRT users would have a sufficient understanding of the 4PL model and the carelessness parameter, and they can estimate item and person parameters under the 4PL model.

A comparison of 3- and 4-option multiple-choice items for medical subspecialty in-training examinations

BackgroundThe American Board of Anesthesiology piloted 3-option multiple-choice items (MCIs) for its 2020 administration of 150-item subspecialty in-training examinations for Critical Care Medicine (ITE-CCM) and Pediatric Anesthesiology (ITE-PA). The 3-option MCIs were transformed from their 4-option counterparts, which were administered in 2019, by removing the least effective distractor. The purpose of this study was to compare physician performance, response time, and item and exam characteristics between the 4-option and 3-option exams.MethodsIndependent-samples t-test was used to examine the differences in physician percent-correct score; paired t-test was used to examine the differences in response time and item characteristics. The Kuder and Richardson Formula 20 was used to calculate the reliability of each exam form. Both the traditional (distractor being selected by fewer than 5% of examinees and/or showing a positive correlation with total score) and sliding scale (adjusting the frequency threshold of distractor being chosen by item difficulty) methods were used to identify non-functioning distractors (NFDs).ResultsPhysicians who took the 3-option ITE-CCM (mean = 67.7%) scored 2.1 percent correct higher than those who took the 4-option ITE-CCM (65.7%). Accordingly, 3-option ITE-CCM items were significantly easier than their 4-option counterparts. No such differences were found between the 4-option and 3-option ITE-PAs (71.8% versus 71.7%). Item discrimination (4-option ITE-CCM [an average of 0.13], 3-option ITE-CCM [0.12]; 4-option ITE-PA [0.08], 3-option ITE-PA [0.09]) and exam reliability (0.75 and 0.74 for 4- and 3-option ITE-CCMs, respectively; 0.62 and 0.67 for 4-option and 3-option ITE-PAs, respectively) were similar between these two formats for both ITEs. On average, physicians spent 3.4 (55.5 versus 58.9) and 1.3 (46.2 versus 47.5) seconds less per item on 3-option items than 4-option items for ITE-CCM and ITE-PA, respectively. Using the traditional method, the percentage of NFDs dropped from 51.3% in the 4-option ITE-CCM to 37.0% in the 3-option ITE-CCM and from 62.7% to 46.0% for the ITE-PA; using the sliding scale method, the percentage of NFDs dropped from 36.0% to 21.7% for the ITE-CCM and from 44.9% to 27.7% for the ITE-PA.ConclusionsThree-option MCIs function as robustly as their 4-option counterparts. The efficiency achieved by spending less time on each item poses opportunities to increase content coverage for a fixed testing period. The results should be interpreted in the context of exam content and distribution of examinee abilities.

Evaluating Modes of Observations Using Hierarchical Signal Detection Rater Models

Scoring of performance can be based on different modes of observation, which can include live and videotaped observations. Although live observations have been the traditional format of measuring examinee performance in education and in healthcare, videotaped observations provide educators and testing agencies the promise of unbiased and standardized evaluations, offering practical solutions to limitations of real-time scoring. This study proposes a measurement model taking into account different modes of observation, using an extension of the hierarchical rater model based on signal detection theory (HRM-SDT). A hierarchical rater model is motivated by the nested structure of scores assigned by raters – scores assigned by raters become indicators of performance quality, which in turn become indicators of examinee ability. This study extends the hierarchical structure of the scoring process to include modes of observation, which serves as an intermediary level between the latent categorical indicator of performance quality and examinee ability, forming a three-level HRM-SDT. Analyses based on real-world data showed differences in the quality of scores from live observations and videotaped recordings. Compared to the traditional HRM-SDT, the overall model fit improved when including modes of observation. Simulations using different sample sizes and conditions provide implications for uses of this model.

An Examination of Individual Ability Estimation and Classification Accuracy Under Rapid Guessing Misidentifications

ABSTRACT To mitigate the deleterious effects of rapid guessing (RG) on ability estimates, several rescoring procedures have been proposed. Underlying many of these procedures is the assumption that RG is accurately identified. At present, there have been minimal investigations examining the utility of rescoring approaches when RG is misclassified, and individual scores are reported. To address this limitation, the present simulation study investigates the effect of RG misclassifications on individual examinee ability estimate bias and classification accuracy when using effort-moderated (EM) scoring. This objective is accomplished by manipulating simulee ability level, RG rate, as well as misclassification type and percentage. Results showed that EM scoring significantly improved ability inferences for examinees engaging in RG; however, the effectiveness of this approach was largely dependent on misclassification type. Specifically, across ability levels, bias tended to be on average lower when falsely classifying effortful responses as RG. Although EM scoring improved bias, it was susceptible to elevated false-positive classifications of ability under high RG.

Outlier Detection Using t-test in Rasch IRT Equating under NEAT Design.

In equating practice, the existence of outliers in the anchor items may deteriorate the equating accuracy and threaten the validity of test scores. Therefore, stability of the anchor item performance should be evaluated before conducting equating. This study used simulation to investigate the performance of the t-test method in detecting outliers and compared its performance with other outlier detection methods, including the logit difference method with 0.5 and 0.3 as the cutoff values and the robust z statistic with 2.7 as the cutoff value. The investigated factors included sample size, proportion of outliers, item difficulty drift direction, and group difference. Across all simulated conditions, the t-test method outperformed the other methods in terms of sensitivity of flagging true outliers, bias of the estimated translation constant, and the root mean square error of examinee ability estimates.

When Should Individual Ability Estimates Be Reported if Rapid Guessing Is Present?

ABSTRACT Testing programs are confronted with the decision of whether to report individual scores for examinees that have engaged in rapid guessing (RG). As noted by the Standards for Educational and Psychological Testing, this decision should be based on a documented criterion that determines score exclusion. To this end, a number of heuristic criteria (e.g., exclude all examinees with RG rates of 10%) have been adopted in the literature. Given that these criteria lack strong methodological support, the objective of this simulation study was to evaluate their appropriateness in terms of individual ability estimate and classification accuracy when manipulating both assessment and RG characteristics. The findings provide evidence that employing a common criterion for all examinees may be an ineffective strategy because a given RG percentage may have differing degrees of biasing effects based on test difficulty, examinee ability, and RG pattern. These results suggest that practitioners may benefit from establishing context-specific exclusion criteria that consider test purpose, score use, and targeted examinee trait levels.

Best Practices for the Independent Medical Examination of Mental and Behavioral Disorders

Abstract Independent medical examinations (IMEs) are elective evaluations of clinical conditions and differ from a clinical consultation in several important ways. They are conducted in many medicolegal contexts and are performed by a regulated health professional who is not the treating health care provider. An IME seeks objective information about the examinee's diagnoses, functional abilities and impairments, and other features relevant to addressing medicolegal-related cases by relying on multiple data sources. Best practice guidelines have been established for the completion of physical IMEs. However, similar mental and behavioral disorder (M&BD) standards are less common, and M&BD IMEs often do not follow a consistent process. This article offers guidance on applying and adapting existing standards to M&BD IMEs. We outlined the appropriate use of psychiatric nomenclature and how to apply recent changes to Chapter 14 of the AMA Guides to the Evaluation of Permanent Impairment. The article further offers a mechanism for introducing greater objectivity into an otherwise subjective process by employing relevant psychological validity testing and conducting a coherence analysis in formulating an opinion. In summary, we provide a synopsis of current best practices and offer the examiner a method for aligning M&BD IMEs with equally high standards of excellence.

Criteria used in setting the English language component of the mature students' entrance examination questions in Ghana

The centrality of examinations to the understanding of competences and abilities of examinees across the globe cannot be overemphasised. The paper focused on exploring the criteria used in setting the English language component of the Mature Students’ Entrance Examination questions in Ghana. The qualitative design (specifically, multiple case study design) was employed for the paper. Using the multi-stage sampling technique, five Ghanaian universities were used for the paper. The data was elicited from in-depth interviews applied on the lecturers who set the English language component of the MSEE questions. [1] data analysis spiral plan was employed to analyse the data. The paper revealed that there were major weaknesses in the criteria used by the universities in areas such as training of the assessors, use of standard assessment models and consideration of test takers’ characteristics. The paper advanced, to policy makers, suggestions such as the adoption of standard assessment models and the training of the assessors for the improvement of the examination2

Preliminary Development of an Item Bank and an Adaptive Test in Mathematical Knowledge for University Students

In the last decades, the development of computerized adaptive testing (CAT) has allowed more precise measurements with a smaller number of items. In this study, we develop an item bank (IB) to generate the adaptive algorithm and simulate the functioning of CAT to assess the domains of mathematical knowledge in Argentinian university students (N=773). Data were analyzed from the Rasch model. A simulation design created with the R software was used to determine the necessary items of the IB to estimate examinee ability. Our results indicate that the IB in the domains of mathematical knowledge is adequate to be applied in CAT, especially to estimate average ability levels. The use of CAT is recommended for rapidly generating indicators of the knowledge acquired by students and to design educational strategies that enhance student performance. Results, constrains, and implications of this study are discussed.

Performance of exploratory structural equation model (ESEM) in detecting differential item functioning

The validity of a standardised test is questioned if an irrelevant construct is accounted for the performance of examinees, which is wrongly modeled as the ability in the construct (test items). A test must ensure precision in the examinee's ability irrespective of their sub-population in any demographic variables. This paper explored the potentials of gender and school location as major covariates on the West African Examinations Council (WAEC) mathematics items among examinees (N=2,866) using Exploratory Structural Equation Modeling (ESEM). The results remarked that the test is multidimensional (six-factors) with compliance fix indices of (χ2 (940)=4882.024, p < 0.05, CFI=0.962, TLI=0.930, RMSEA=0.038, SRMR=0.030, 90 % CI=0.037-0.039, Akaike information criterion (AIC)=147290.577, Bayesian information criterion (BIC)=149585.436 and Sample-size adjusted BIC=148362.154) respectively. Also, there were 10 (20 %) significant DIF items in the WAEC to gender, while 3 (6 %) of the items indicated significant DIF to school location. Observed DIF items acquaint test developers; the existence of DIF may differentially affect the performance of examinees with the same ability level. The implications of the test are severe for the examinees. Hence, accurate and unbiased assessment should be the basic principles for any test item measurement, and test developers need to test the items to be free from biases psychometrically.

Detection of Outliers in Anchor Items Using Modified Rasch Fit Statistics

ABSTRACT The existence of outliers in the anchor items can be detrimental to the estimation of examinee ability and undermine the validity of score interpretation across forms. However, in practice, anchor item performance can become distorted due to various reasons. This study compares the performance of modified INFIT and OUTFIT Rasch statistics with the Logit Difference approach with 0.3 and 0.5 as the predetermined cutoff values, and the Robust z statistic with 1.645 and 2.7 as the cutoff values through a simulation study by varying the sample size, proportion of outliers, item difficulty drift direction, and group difference magnitude. The results suggest that both modified INFIT and OUTFIT statistics perform very similarly and outperform the other methods in all aspects, including sensitivity of flagging outliers, specificity of flagging non-outliers, recovery of translation constant, and recovery of examinee ability in all simulated conditions.

Cut-Score Operating Function Extensions: Penalty-Based Errors and Uncertainty in Standard Settings.

We model pass/fail examinations aiming to provide a systematic tool to minimize classification errors. We use the method of cut-score operating functions to generate specific cut-scores on the basis of minimizing several important misclassification measures. The goal of this research is to examine the combined effects of a known distribution of examinee abilities and uncertainty in the standard setting on the optimal choice of the cut-score. In addition, we describe an online application that allows others to utilize the cut-score operating function for their own standard settings.

A multidimensional generalized many-facet Rasch model for rubric-based performance assessment

Performance assessment, in which human raters assess examinee performance in a practical task, often involves the use of a scoring rubric consisting of multiple evaluation items to increase the objectivity of evaluation. However, even when using a rubric, assigned scores are known to depend on characteristics of the rubric’s evaluation items and the raters, thus decreasing ability measurement accuracy. To resolve this problem, item response theory (IRT) models that can estimate examinee ability while considering the effects of these characteristics have been proposed. These IRT models assume unidimensionality, meaning that a rubric measures one latent ability. In practice, however, this assumption might not be satisfied because a rubric’s evaluation items are often designed to measure multiple sub-abilities that constitute a targeted ability. To address this issue, this study proposes a multidimensional IRT model for rubric-based performance assessment. Specifically, the proposed model is formulated as a multidimensional extension of a generalized many-facet Rasch model. Moreover, a No-U-Turn variant of the Hamiltonian Markov chain Monte Carlo algorithm is adopted as a parameter estimation method for the proposed model. The proposed model is useful not only for improving the ability measurement accuracy, but also for detailed analysis of rubric quality and rubric construct validity. The study demonstrates the effectiveness of the proposed model through simulation experiments and application to real data.

The Effect of Multiple-Choice Test Items’ Difficulty Degree on the Reliability Coefficient and the Standard Error of Measurement Depending on the Item Response Theory (IRT)

This study aims at identifying the effect of multiple-choice test items' difficulty degree on the reliability coefficient and the standard error of measurement depending on the item response theory IRT. To achieve the objectives of the study, (WinGen3) software was used to generate the IRT parameters (difficulty, discrimination, guessing) for four forms of the test. Each form consisted of (30) items with different difficulty coefficients averages (-0.24, 0.24, 0.42, 0.93). The resulting items parameters were utilized to generate the ability and responses of (3000) examinees based on the three-parameter model. These data were converted into a readable file using the (SPSS) and the (BILOG-MG3) software. Then the reliability coefficients for the four test forms, the items parameters, and the items information function were calculated, and dependence on the information function values to calculate the standard error of measurement for each item.The results of the study showed that there are statistically significant differences at the level of significance (α ≤ 0.05) between the averages of the values of the standard error of measurement attributed to the difference in the difficulty degree of the items in favor of the test with the higher difficulty coefficient. The results also found that there are apparent differences between the test reliability parameters attributed to the difficulty degree of the test according to the three-parameter model in favor of the form with the average difficulty degree.