Item Parameters Research Articles

Examining the influencing factors and the functioning mechanism of online learning supporting learners' empowerment.

The construction of an effective technology-based environment to promote learning and enable online learning to truly empower learners has become an increasingly important issue of concern. To some extent, an in-depth analysis of the key influencing factors and their functioning mechanism of online learning supporting learners' empowerment may pave a new way for the understanding of this issue. In view of this, this study, grounded on the reciprocal determinism theory, explored the influencing paths among environment elements, online behaviors, person factors, and learners' empowerment by using structural equation modeling (SEM) approach, and examined the structure effect relationships among these dimensions. A total of 396 Chinese university students reported on their perceptions of environment elements, online behaviors, person factors, and learners' empowerment through the questionnaire survey. The results showed that there was a strong correlation between the three dimensions of the reciprocal determinism theory, and the three dimensions all exerted a positive effect on learners' empowerment. Among them, the item parameter estimates revealed that online environment, online interaction, and online cooperation were significant in explaining the influence of environment elements and online behaviors on learners' empowerment. Based on the empirical results, this study provided some implications and suggestions for the rationality of the structure devising of online learning environment, the effectiveness of online teacher-student interaction, and the development of students' learning autonomy in technology-based learning. Thus, this study helped understand the specific contribution of external and internal factors to learners' empowerment of using technology for online learning and would inform the construction of the optimized online learning environments and teacher-student relationships to enhance students' sense of online presence and learning initiative.

Rasch 모형을 적용한 문항분석 및 차별기능문항 탐색

In this study, the results of 597 subjects who took the Physics Certification Exam level 1-2 conducted in February 2021 were reviewed. Using an analysis method based on the item response theory(IRT) for research, the fairness and reliability of the test were considered. To analyze the item and test fit, it was employed the Rash model which estimates only the item difficulty on the IRT. Moreover the gender-based differential item function(DIF) was assessed using the WINSTEPS program(Ver. 4.3). As a result, the individual items and the overall test fit seemed quite sustainable, however in the view point of the DIF detection, two items with each gender-favorable answers were found with statistical test and actual effect size. Conclusively, it is needed to consider the effects of the composition of the options and items, clarity of the item content to keep the invariance of the item parameter estimation for the gender-independent measurement. Through continuous DIF detection, it was predicted that not only the fairness, consistency, and reliability of the inspection, but also the possibility of quality test items management.

Testing beat perception without sensory cues to the beat: the Beat-Drop Alignment Test (BDAT)

Beat perception can serve as a window into internal time-keeping mechanisms, auditory–motor interactions, and aspects of cognition. One aspect of beat perception is the covert continuation of an internal pulse. Of the several popular tests of beat perception, none provide a satisfying test of this faculty of covert continuation. The current study proposes a new beat-perception test focused on covert pulse continuation: The Beat-Drop Alignment Test (BDAT). In this test, participants must identify the beat in musical excerpts and then judge whether a single probe falls on or off the beat. The probe occurs during a short break in the rhythmic components of the music when no rhythmic events are present, forcing participants to judge beat alignment relative to an internal pulse maintained in the absence of local acoustic timing cues. Here, we present two large (N > 100) tests of the BDAT. In the first, we explore the effect of test item parameters (e.g., probe displacement) on performance. In the second, we correlate scores on an adaptive version of the BDAT with the computerized adaptive Beat Alignment Test (CA-BAT) scores and indices of musical experience. Musical experience indices outperform CA-BAT score as a predictor of BDAT score, suggesting that the BDAT measures a distinct aspect of beat perception that is more experience-dependent and may draw on cognitive resources such as working memory and musical imagery differently than the BAT. The BDAT may prove useful in future behavioral and neural research on beat perception, and all stimuli and code are freely available for download.

Development of a High-Accuracy and Effective Online Calibration Method in CD-CAT Based on Gini Index

To improve the calibration accuracy and calibration efficiency of cognitive diagnostic computerized adaptive testing (CD-CAT) for new items and, ultimately, contribute to the widespread application of CD-CAT in practice, the current article proposed a Gini-based online calibration method that can simultaneously calibrate the Q-matrix and item parameters of new items. Three simulation studies with simulated and real item banks were conducted to investigate the performance of the proposed method and compare it with the joint estimation algorithm (JEA) and the single-item estimation (SIE) methods. The results indicated that the proposed Gini-based online calibration method yielded higher calibration efficiency than those of the SIE method and outperformed the JEA method on item calibration tasks in terms of both accuracy and efficiency under most experimental conditions.

The study of the effect of item parameter drift on ability estimation obtained from adaptive testing under different conditions

Item parameter drift (IPD) is the systematic differentiation of parameter values of items over time due to various reasons. If it occurs in computer adaptive tests (CAT), it causes errors in the estimation of item and ability parameters. Identification of the underlying conditions of this situation in CAT is important for estimating item and ability parameters with minimum error. This study examines the measurement precision of IPD and its impacts on the test information function (TIF) in CAT administrations. This simulation study compares sample size (1000, 5000), IPD size (0.00 logit, 0.50 logit, 0.75 logit, 1.00 logit), percentage of items containing IPD (0%, 5%, 10%, 20%), three time points and item bank size (200, 500, 1000) conditions. To examine the impacts of the conditions on ability estimations; measurement precision, and TIF values were calculated, and factorial analysis of variance (ANOVA) for independent samples was carried out to examine whether there were any differences between estimations in terms of these factors. The study found that an increase in the number of measurements using item bank with IPD items results in a decrease in measurement precision and the amount of information the test provides. Factorial ANOVA for independent samples revealed that measurements precision and TIF differences are mostly statistically significant. Although all IPD conditions negatively affect measurement precision and TIF, it has been shown that sample size and item bank size generally do not have an increasing or decreasing effect on these factors.

PSYCHOMETRIC ANALYSIS OF SENIOR SECONDARY SCHOOL CERTIFICATE EXAMINATION (SSCE) 2017 NECO ENGLISH LANGUAGE MULTIPLE CHOICE TEST ITEM IN KWARA STATE USING ITEM RESPONSE THEORY

The study determined the dimensionality of 2017 National Examinations Council (NECO) English Language multiple-choice test item and estimated the item parameter indices (discrimination, difficulty, guessing and carelessness) using four parameter logistic model. The ex-post facto design was employed for the study. The population for the study comprised all candidates/test-takers who enrolled and sat for June/July Senior School Certificate Examination (SSCE) 2017 NECO English Language Examination in Kwara State, Nigeria with 12,000 samples purposively selected from sixteen Local Government Area in the State. The research instruments used for the study were Optical Marks Record Sheets for the NECO June/July 2017 English Language objectives items. The responses of the testees were scored dichotomously. The data collected were calibrated using four parameters logistic model. The results showed that the 2017 English Language multiple-choice item among SSCE students in Kwara State does not violate the assumption of unidimensionality. Also, the results showed that only two items (i.e., items 11 and 16) were able to satisfy the rule of thumb while the remaining 98 items were poor because they violated the rule of thumb in one way or the other.

Caregiver-reported newborn term and preterm motor abilities: psychometrics of the PediaTracTM Motor domain.

Approximately 5-10% of children exhibit developmental deviations in motor skills or other domains; however, physicians detect less than one-third of these abnormalities. Systematic tracking and early identification of motor deviations are fundamental for timely intervention. Term and preterm neonates were prospectively assessed at the newborn (NB) period in a study of the psychometric properties of the Motor (MOT) domain of PediaTracTM v3.0, a novel caregiver-based development tracking instrument. Item response theory graded response modeling was used to model item parameters and estimate theta, an index of the latent trait, motor ability. Exploratory factor analysis (EFA) was conducted to examine the dimensionality and factor structure. In a cohort of 571 caregiver/infant dyads (331 term, 240 preterm), NB MOT domain reliability was high (rho = 0.94). Item discrimination and item difficulty of each of the 15 items could be reliably modeled across the range of motor ability. EFA confirmed that the items constituted a single dimension with second-order factors, accounting for 43.20% of variance. The latent trait, motor ability, could be reliably estimated at the NB period. The caregiver-reported Motor domain of PediaTrac provides a reliable estimate of the latent trait of motor ability during the newborn period. This is the first known caregiver-reported instrument that can assess motor ability in the newborn period with high reliability in term and preterm infants. Item response theory methods were employed that will allow for future characterization of developmental subgroups and motor trajectories. The PediaTrac Motor domain can support early identification of at-risk infants. Including caregivers in digital reporting and child-centered monitoring of motor functioning may improve access to care.

Using Mokken scaling techniques to explore carelessness in survey research.

Careless responding is a pervasive issue that impacts the interpretation and use of responses from survey instruments. Researchers have proposed numerous useful methods for detecting carelessness in survey research, including relatively simple summary statistics such as the frequency of adjacent responses in the same category (e.g., "long-string" analysis) and outlier statistics (e.g., Mahalanobis distance). Researchers have also used methods based on item response theory (IRT) models to identify examinees whose response patterns are unexpected given item parameters. However, researchers have not fully considered the use of nonparametric IRT methods based on Mokken scale analysis (MSA) to detect carelessness in survey research. MSA is a promising framework in which to consider participant carelessness because it is well suited to contexts in which parametric IRT models may not be appropriate, while still maintaining a focus on fundamental measurement requirements. We used a real data analysis and a simulation study to examine the sensitivity of MSA indicators of response quality to examinee carelessness and compared the results to those from standalone indicators. We also examined the impact of carelessness on the sensitivity of MSA item quality indicators. Numeric and graphical indicators of response quality from MSA indicators were sensitive to examinee carelessness. Graphical displays of nonparametric person response functions (PRFs) provided supplementary insight that can alert researchers to potentially problematic responses. Our results also indicated that MSA indicators of item quality are robust to the presence of participant carelessness. We consider the implications of our findings for research and practice.

Item parameter estimations for multidimensional graded response model under complex structures

Item parameter recovery in the compensatory multidimensional graded response model (MGRM) under simple and complex structures with rating-scale item response data was examined. A simulation study investigated factors that influence the precision of item parameter estimation, including sample size, intercorrelation between the dimensions, and test lengths for the MGRM under balanced and unbalanced complex structures, as well as the simple structure. The item responses for the MGRM were generated and analyzed across conditions using the R package mirt. The bias and root mean square error (RMSE) was used to evaluate item parameter recovery. Results suggested that item parameter estimation was more accurate in balanced complex structure conditions than in unbalanced or simple structures, especially when the test length was 40 items, and the sample size was large. Further, the mean bias and RMSE in the recovery of item threshold estimates along the two dimensions for both balanced and unbalanced complex structures were consistent across all conditions.

Executive Function Touch battery: Translation and preliminary measure validation for Pakistani preschoolers.

Keeping in mind the importance of measuring early executive function (EF) skills in low and middle-income countries, the present study examined the feasibility and preliminary psychometric properties of a performance-based computerized EF measure; EF Touch, to be used with Pakistani preschoolers. Review of the content and Urdu translation of verbal instruction EF Touch battery was carried out by subject matter experts before data collection from the 120 preschoolers aged between 3.1 to 5.9 years. The feasibility report indicated that between 79.2% -100% of the preschoolers completed each executive function task. Confirmatory factor analysis revealed the unidimensionality of the EF battery. Item response theory models were used for the initial assessment of tasks and item parameters. Results demonstrated that each task worked invariantly across subgroups of preschoolers residing in low and middle-income households. Moreover, preschoolers showed differences on each task, and task scores reflect their latent EF skills in the low to moderate range. The battery was demonstrated as a feasible and reliable measure for use with low and middle-income countries specifically in Pakistan.

Evaluating Gender Differences in Problematic Smartphone Use

Abstract: The Smartphone Addiction Inventory (SPAI) is widely used to measure problematic smartphone use (PSU). Although the SPAI has been translated and validated in different countries, its measurement invariance across gender has received little research attention. This study aimed to examine whether men and women interpreted the Italian version of the SPAI (SPAI-I) similarly and, consequently, whether the observed gender differences in SPAI scores, which have been shown in previous studies, could be due to true differences, rather than to differences in measurement. Six hundred nineteen Italian young adults ( Mage = 22.02 ± 2.63; 55.7% women) took part in the study and completed the SPAI-I. Multigroup CFA was applied to test the measurement invariance across gender, and the item parameter invariance was investigated with the item-response theory (IRT) differential item functioning (DIF) method for multidimensional models. Evidence of measurement invariance across gender was found. Only one item (i.e., item 14, “The idea of using smartphone comes as the first thought on mind when waking up each morning”) out of 24 items of the SPAI-I showed DIF with a large effect size. Gender-related differences found with the SPAI-I reflect true differences in smartphone overuse rather than specific characteristics of the measure.

On-the-fly parameter estimation based on item response theory in item-based adaptive learning systems.

Online learning systems are able to offer customized content catered to individual learner's needs, and have seen growing interest from industry and academia alike in recent years. In contrast to the traditional computerized adaptive testing setting, which has a well-calibrated item bank with new items added periodically, the online learning system has two unique features: (1) the number of items is large, and they have likely not gone through costly field testing for item calibration; and (2) the individual's ability may change as a result of learning. The Elo rating system has been recognized as an effective method for fast updating of item and person parameters in online learning systems to enable personalized learning. However, the updating parameter in Elo has to be tuned post hoc, and Elo is only suitable for the Rasch model. In this paper, we propose the use of a moment-matching Bayesian update algorithm to estimate item and person parameters on the fly. With sequentially updated item and person parameters, a modified maximum posterior weighted information criterion (MPWI) is proposed to adaptively assign items to individuals. The Bayesian updated algorithm along with MPWI is validated in a simulated multiple-session online learning setting, and the results show that the new combo can achieve fast and reasonably accurate parameter estimations that are comparable to random selection, match-difficulty selection, and traditional online calibration. Moreover, the combo can still function reasonably well with as low as 20% of items being pre-calibrated in the item bank.

Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items.

In traditional test models, test items are independent, and test-takers slowly and thoughtfully respond to each test item. However, some test items have a common stimulus (dependent test items in a testlet), and sometimes test-takers lack motivation, knowledge, or time (speededness), so they perform rapid guessing (RG). Ignoring the dependence in responses to testlet items can negatively bias standard errors of measurement, and ignoring RG by fitting a simpler item response theory (IRT) model can bias the results. Because computer-based testing captures response times on testlet responses, we propose a mixture testlet IRT model with item responses and response time to model RG behaviors in computer-based testlet items. Two simulation studies with Markov chain Monte Carlo estimation using the JAGS program showed (a) good recovery of the item and person parameters in this new model and (b) the harmful consequences of ignoring RG (biased parameter estimates: overestimated item difficulties, underestimated time intensities, underestimated respondent latent speed parameters, and overestimated precision of respondent latent estimates). The application of IRT models with and without RG to data from a computer-based language test showed parameter differences resembling those in the simulations.

Efficient Metropolis-Hastings Robbins-Monro Algorithm for High-Dimensional Diagnostic Classification Models.

The expectation-maximization (EM) algorithm is a commonly used technique for the parameter estimation of the diagnostic classification models (DCMs) with a prespecified Q-matrix; however, it requires O(2 K ) calculations in its expectation-step, which significantly slows down the computation when the number of attributes, K, is large. This study proposes an efficient Metropolis-Hastings Robbins-Monro (eMHRM) algorithm, needing only O(K + 1) calculations in the Monte Carlo expectation step. Furthermore, the item parameters and structural parameters are approximated via the Robbins-Monro algorithm, which does not require time-consuming nonlinear optimization procedures. A series of simulation studies were conducted to compare the eMHRM with the EM and a Metropolis-Hastings (MH) algorithm regarding the parameter recovery and execution time. The outcomes presented in this article reveal that the eMHRM is much more computationally efficient than the EM and MH, and it tends to produce better estimates than the EM when K is large, suggesting that the eMHRM is a promising parameter estimation method for high-dimensional DCMs.

Effect of Item Parameter Drift in Mixed Format Common Items on Test Equating

The aim of the study was to examine the common items in the mixed format (e.g., multiple-choices and essay items) contain parameter drifts in the test equating processes performed with the common item non-equivalent groups design. In this study, which was carried out using Monte Carlo simulation with a fully crossed design, the factors of test length (30 and 50), sample size (1000 and 3000), common item ratio (30 and 40%), ratio of items with item parameter drift (IPD) in common items (20 and 30%), location of common items in tests (at the beginning, randomly distributed, and at the end) and IPD size in multiple-choice items (low [0.2] and high [1.0]) were studied. Four test forms were created, and two test forms do not contain parameter drifts. After the parameter drift was performed on the first of the other two test forms, the parameter drift was again performed on the second test form. Test equating results were compared using the root mean squared error (RMSE) value. As a result of the research, ratio of items with IPD in common items, IPD size in multiple-choice items, common item ratio, sample size and test length on equating errors were found to be significant.

Psychometric Evaluation of the Korean Version of PROMIS Self-Efficacy for Managing Symptoms Item Bank: Item Response Theory

To evaluate the psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS) self-efficacy for managing symptoms of the version 1.0 item bank in Korea. This study consisted of two phases: first, developing the Korean version of the item bank following the translation guidelines; and second, performing a cross-sectional study to evaluate its psychometric properties using the item response theory. This study enrolled 323 patients with type 2 diabetes mellitus between July and August 2020. Cronbach's α was used to assess the reliability of this item bank. Confirmatory factor analysis, using diagonally weighted least squares, was used to identify the assumptions of item response theory. Item parameter estimates including discrimination and thresholds were derived using the graded response model of the item response theory to reflect patient-reported outcomes as individualized responses. The Korean version of the item bank demonstrated good reliability (Cronbach's α=.98) and its discrimination ranged from 1.82 to 4.93. The thresholds resulted in the establishment of a category response curve for each item. However, no overlap was observed among the category curves. Moreover, the differential item functioning was not significant for age, gender, and income variables. The graded response model and differential item functioning provided qualitative evidence that demonstrated acceptable psychometric properties of symptom management self-efficacy among patients. This item bank is expected to provide adequate assessments of self-efficacy of symptom management for patients with a chronic disease, which can contribute to nursing research and intervention.

Towards standardization of measuring anxiety and depression: Differential item functioning for language and Dutch reference values of PROMIS item banks.

The outcomes anxiety and depression are measured frequently by healthcare providers to assess the impact of a disease, but with numerous instruments. PROMIS item banks provide an opportunity for standardized measurement. Cross-cultural validity of measures and the availability of reference values are prerequisites for standardized measurement. PROMIS Anxiety and Depression item banks were completed by 1002 representative Dutch persons. To evaluate cross-cultural validity, data from US participants in PROMIS wave 1 were used and differential item functioning (DIF) was investigated, using an iterative hybrid of logistic regression and item response theory. McFadden's pseudo R2-change of 2% was the critical threshold. The impact of any DIF on full item banks and short forms was investigated. To obtain Dutch reference values, T-scores for anxiety and depression were calculated for the complete Dutch sample, and age-group and gender subpopulations. Thresholds corresponding to normal limits, mild, moderate and severe symptoms were computed. In both item banks, two items had DIF but with minimal impact on population level T-scores for full item banks and short forms. The Dutch general population had a T-score of 49.9 for anxiety and 49.6 for depression, similar to the T-scores of 50.0 of the US general population. T-scores for age-group and gender subpopulations were also similar to T-scores of the US general population. Thresholds for mild, moderate and severe anxiety and depression were set to 55, 60 and 70, identical to US thresholds. The limited number of items with DIF and its minimal impact, enables the use of standard (US) item parameters and comparisons of scores between Dutch and US populations. The Dutch reference values provide an important tool for healthcare professionals and researchers to evaluate and interpret symptoms of anxiety and depression, stimulating the uptake of PROMIS measures, and contributing to standardized outcome measurement.

Relative Robustness of CDMs and (M)IRT in Measuring Growth in Latent Skills.

Previous studies have demonstrated evidence of latent skill continuity even in tests intentionally designed for measurement of binary skills. In addition, the assumption of binary skills when continuity is present has been shown to potentially create a lack of invariance in item and latent ability parameters that may undermine applications. In this article, we examine measurement of growth as one such application, and consider multidimensional item response theory (MIRT) as a competing alternative. Motivated by prior findings concerning the effects of skill continuity, we study the relative robustness of cognitive diagnostic models (CDMs) and (M)IRT models in the measurement of growth under both binary and continuous latent skill distributions. We find CDMs to be a less robust way of quantifying growth under misspecification, and subsequently provide a real-data example suggesting underestimation of growth as a likely consequence. It is suggested that researchers should regularly attend to the assumptions associated with the use of latent binary skills and consider (M)IRT as a potentially more robust alternative if unsure of their discrete nature.

Evaluating the impact of calibration of patient-reported outcomes measures on results from randomized clinical trials: a simulation study based on Rasch measurement theory

BackgroundMeaningfully interpreting patient-reported outcomes (PRO) results from randomized clinical trials requires that the PRO scores obtained in the trial have the same meaning across patients and previous applications of the PRO instrument. Calibration of PRO instruments warrants this property. In the Rasch measurement theory (RMT) framework, calibration is performed by fixing the item parameter estimates when measuring the targeted concept for each individual of the trial. The item parameter estimates used for this purpose are typically obtained from a previous “calibration” study. But imposing this constraint on item parameters, instead of freely estimating them directly in the specific sample of the trial, may hamper the ability to detect a treatment effect. The objective of this simulation study was to explore the potential negative impact of calibration of PRO instruments that were developed using RMT on the comparison of results between treatment groups, using different analysis methods.MethodsPRO results were simulated following a polytomous Rasch model, for a calibration and a trial sample. Scenarios included varying sample sizes, with instrument of varying number of items and modalities, and varying item parameters distributions. Different treatment effect sizes and distributions of the two patient samples were also explored. Cross-sectional comparison of treatment groups was performed using different methods based on a random effect Rasch model. Calibrated and non-calibrated approaches were compared based on type-I error, power, bias, and variance of the estimates for the difference between groups.ResultsThere was no impact of the calibration approach on type-I error, power, bias, and dispersion of the estimates. Among other findings, mistargeting between the PRO instrument and patients from the trial sample (regarding the level of measured concept) resulted in a lower power and higher position bias than appropriate targeting.ConclusionsCalibration does not compromise the ability to accurately assess a treatment effect using a PRO instrument developed within the RMT paradigm in randomized clinical trials. Thus, given its essential role in producing interpretable results, calibration should always be performed when using a PRO instrument developed using RMT as an endpoint in a randomized clinical trial.

Imputation of missing values within WHODAS 2.0 data collected from low back pain patients using the response function approach

Purpose To investigate the impact of missing data and imputation with the response function (RF) approach on bias and precision of disability estimates as well as reliability of scale of WHO Disability Assessment Schedule 2.0. Material and methods Data were collected by face-to-face interviews and self-report surveys from 284 respondents with low back pain. Hypothetical datasets were created by using person and item parameters of real data. A simulation study was devised to assess the ability parameters and reliability measures on incomplete and imputed datasets. Rasch model was used to evaluate latent trait levels. Imputation was carried out using the response function method. Results Almost the same level of bias and MSE was reached. While the missing rate increases, the Person separation index slightly reduced, still exceeded 0.94 and Cronbach alpha values have similar mean values of 0.99 with larger variations. After deletion of four items of "work or school activities" in domain 5, reliability measures reduced the lowest. Conclusion Construct validity is preserved. Problems regarding the compliance of the items with the target group still persist. When researchers encounter missingness in data collected with WHODAS 2.0, the response function can be usefully implemented to impute missing values to improve the reliability of disability level estimates. Implications for rehabilitation The Turkish version of the 36-item WHODAS is reliable and valid for assessing health and disability status in patients with low back pain. A critical issue is a systematic nonresponse was highly observed in items related to “work/school life activities” which are allowed to be skipped and excluded from the scoring according to WHODAS 2.0 training manual. Imputation of missing values within four items of "work or school activities" in domain 5 using the response function approach showed satisfactory reliability for the estimation of disability in adults with low back pain. This study showed missing data imputations with response function can be implemented by a statistician as contribute to a missing data management process better tailored to clinicians' interpretations.