Classical Test Theory Framework Research Articles

고전검사이론과 문항반응이론으로 살펴본 모의 TOEIC 문항분석

This study analyzed student test performance on the mock TOEIC in order to investigate the validity and reliability of the test items within a framework of Classical Test Theory (CTT) and Item Response Theory (IRT). The test, including 49 listening and 51 reading multiple-choice items, was administered to a total of 98 registered university students. Their responses to each test item were statistically analyzed with respect to four indices: item difficulty, item discrimination, item guessing and distractor’s attractiveness. For the analysis, each item was analyzed by applying CTT and IRT according to a 3-parameter logistic model with Pearson correlation coefficient and Kuder Richardson-20 (KR-20). According to the item analysis, it was found that the test displayed a high level of reliability of .920, and the difficulty level was .563, indicating that it was in the desirable range between .25 and .75. The mean discrimination power was also .368, which is within the desirable range of over .3. However, further statistical and qualitative item analyses found that the mock TOEIC contained test items within an inappropriate difficulty level, low item discrimination and implausible distractors. Therefore, this study provides suggestions for enhancing the quality of the mock TOEIC, which can also apply to future TOEICs.

The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment.

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a "Reliability Factor" that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

An Item Analysis of the Mock CSAT English Test

This study analyzed students’ test performance on the mock College Scholastic Ability Test (CSAT) English test to examine the reliability and validity of the test items within a framework of Classical Test Theory (CTT). The test used for the present study was the mock CSAT English test administered in June 2021 and item difficulty data was collected from 100 3rd grade high school students who took the test. For the item analysis, Test Analysis Program (TAP), a computational tool was used to investigate the extent to which the mock CSAT is a valid test based on item difficulty, item discrimination and distractors’ attractiveness. According to the item analysis, it was found that the reliability of the test was as high as .876, which is reliable and the level of difficulty was .713, indicating that it wasin the desirable range between .25 and .75. The mean discrimination power was .337, which is within the desirable range of over .3. However, an imbalance of item difficulty between the listening and reading section were detected with unreasonably weighted items and implausible distractors. Therefore, this study provides implications for improvement for the mock CSAT, which can also apply to future CSATs.

Patient-Reported Outcome Measures in Symptomatic, Non-Limb-Threatening Peripheral Artery Disease: A State-of-the-Art Review.

Patient-reported outcome measures (PROMs) are health outcomes directly reported by the patient that can be used to measure the effect of disease and treatments on patient perceived well-being. This review summarizes current evidence regarding the validation of PROMs in people with symptomatic, nonlimb-threatening peripheral artery disease. A literature search was conducted to identify studies of symptomatic peripheral artery disease without limb-threatening ischemia that included PROMs and had sample sizes ≥25. PROMs were summarized along a continuum of validation using classical test theory framework and according to whether they fulfilled defined criteria for (1) content validity; (2) psychometric validation; and (3) further validation evidence base expansion. Of 2198 articles identified, 157 (7.1%) met inclusion criteria. Twenty-four PROMs in patients with symptomatic peripheral artery disease were reviewed. Among disease-specific PROMs, 8 of 15 had excellent reliability as measured by a Cronbach alpha ≥0.80. Based on established criteria for PROM responsiveness, 6 of 15 disease-specific PROMs demonstrated excellent sensitivity to change. Of these, the disease-specific peripheral artery questionnaire, vascular quality of life questionnaire, and walking impairment questionnaire met criteria for validation at each stage of the continuum. For generic (nondisease specific) PROMs, the European Quality of Life 5-Dimension and SF-36 had the most extensive evidence of validation. Evidence from this review can inform selection of PROMs aligned with scientific and clinical goals, given the variable degree of validation and potential complementary nature of the measures.

Measuring COVID-19 Related Anxiety in Parents: Psychometric Comparison of Four Different Inventories.

BackgroundThe COVID-19 outbreak and the measures to contain the global pandemic can have an impact on the well-being and mental health status of individuals. Parents of young children are particularly at risk for high levels of parental stress due to the current public health crisis, which can impact parenting behaviors and children’s well-being. Although different initial scales have been developed to measure COVID-19–related anxiety, they have not yet been tested sufficiently in parent samples. A brief measure of COVID-19–related anxiety is necessary for both quick assessment in practice and in larger epidemiological studies of parents.ObjectiveThe purpose of this study is to compare the distributions, validities, and reliabilities of four different COVID-19 anxiety scales: Fear of COVID-19 Scale, Coronavirus Anxiety Scale, Pandemic Anxiety Scale, and one subscale of the COVID Stress Scales. Based on the psychometric properties of these scales, we aim to provide recommendations for a brief unidimensional inventory to assess COVID-19–related anxiety among parents.MethodsA cross-sectional web-based survey of 515 German-speaking parents (465 mothers, 90.3%) with at least one child aged 0-6 years was conducted during a 6-week period (June 29 to August 9, 2020). Half of the parents were recruited via Facebook parenting groups, while the other half were recruited through childcare centers. We psychometrically tested 25 items on COVID-19–related anxiety using the framework of classical test theory, including item analysis, correlational analysis of family variables, and exploratory factor analysis. Moreover, an item response theory approach was applied to estimate item discriminations and item difficulties.ResultsBased on the psychometric properties, three items of the Pandemic Anxiety Scale were identified as a single unidimensional factor. The adapted scale demonstrated acceptable internal consistency (α=.79), moderate to high item discrimination, strong positive intercorrelation with two other COVID-19 anxiety scales, and a small positive association with parenting stress. Mothers and fathers did not differ in total scores (t513=−0.79, P=.42).ConclusionsFactor analysis suggests that existing COVID-19–related anxiety scales measure different latent constructs of anxiety. Furthermore, all scales showed only small to moderate correlations with trait health anxiety, suggesting that COVID-19–related anxiety is distinct from general health anxiety. The adapted “disease anxiety” subscale of the Pandemic Anxiety Scale is an economical measure for assessing COVID-19–related anxiety in parents. Directions for future research are outlined.

Illustration of Multilevel Explanatory IRT Model DIF Testing with the Creative Thinking Scale

ABSTRACTCreativity has been well studied in the past several decades, and numerous measures have been developed to assess creativity. However, validity evidence associated with each measure is often mixed. In particular, the social consequence aspect of validity has received little attention. This is partly due to the difficulty of testing for differential item functioning (DIF) within the traditional classical test theory framework, which still remains the most popular approach to assessing creativity. Hence, this study provides an example of examining differential item functioning using multilevel explanatory item response theory models. The Creative Thinking Scale was tested for DIF in a sample of 1043 10th–12th graders. Results revealed significant uniform and non‐uniform DIF for some items. Differentially functioning items are able to produce measurement bias and should be either deleted or modeled. The detailed implications for researchers and practitioners are discussed.

Exploration of Factors Affecting the Added Value of Test Subscores

ABSTRACTRecently, interest in test subscore reporting for diagnosis purposes has been growing rapidly. The two simulation studies here examined factors (sample size, number of subscales, correlation between subscales, and three factors affecting subscore reliability: number of items per subscale, item parameter distribution, and data generating model) that affected the value of reporting subscores within the classical test theory framework. Results showed that a higher proportion of subscores of added value was related to lower correlation between subscales, more items per subscale, no guessing in responses, smaller variability in difficulty parameters, and matched average item difficulty and average examinee ability.

Classical Test Theory and Item Response Theory in Measuring Validity of Peer-Grading in Massive Open Online Courses

The article presents the results of research on validity of peer-review assignments in massive open online courses within the framework of classical test theory (CTT) and item response theory (IRT). CTT-based analysis yielded data on convergent validity of the peer-review assignment, the low level of its criterion validity, and rater disagreement. IRT-based analysis revealed rater bias and established that experts largely tend to be lenient and overrate their peers. The findings are used to discuss the advantages and disadvantages of the psychometric theories in question and the opportunities for combining the two.

The 12-item Self-Report World Health Organization Disability Assessment Schedule (WHODAS) 2.0 Administered Via the Internet to Individuals With Anxiety and Stress Disorders: A Psychometric Investigation Based on Data From Two Clinical Trials.

BackgroundThe World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is a widespread measure of disability and functional impairment, which is bundled with the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) for use in psychiatry. Administering psychometric scales via the Internet is an effective way to reach respondents and allow for convenient handling of data.ObjectiveThe aim was to study the psychometric properties of the 12-item self-report WHODAS 2.0 when administered online to individuals with anxiety and stress disorders. The WHODAS 2.0 was hypothesized to exhibit high internal consistency and be unidimensional. We also expected the WHODAS 2.0 to show high 2-week test-retest reliability, convergent validity (correlations approximately .50 to .90 with other self-report measures of functional impairment), that it would differentiate between patients with and without exhaustion disorder, and that it would respond to change in primary symptom domain.MethodsWe administered the 12-item self-report WHODAS 2.0 online to patients with anxiety and stress disorders (N=160) enrolled in clinical trials of cognitive behavior therapy, and analyzed psychometric properties within a classical test theory framework. Scores were compared with well-established symptom and disability measures, and sensitivity to change was studied from pretreatment to posttreatment assessment.ResultsThe 12-item self-report WHODAS 2.0 showed high internal consistency (Cronbach alpha=.83-.92), high 2-week test-retest reliability (intraclass correlation coefficient=.83), adequate construct validity, and was sensitive to change. We found preliminary evidence for a three-factorial structure, but one strong factor accounted for a clear majority of the variance.ConclusionsWe conclude that the 12-item self-report WHODAS 2.0 is a psychometrically sound instrument when administered online to individuals with anxiety and stress disorders, but that it is probably fruitful to also report the three subfactors to facilitate comparisons between studies.Trial RegistrationClinicaltrials.gov NCT02540317; https://clinicaltrials.gov/ct2/show/NCT02540317 (Archived by WebCite at http://www.webcitation.org/6vQEdYAem); Clinicaltrials.gov NCT02314065; https://clinicaltrials.gov/ct2/show/NCT02314065 (Archived by WebCite at http://www.webcitation.org/6vQEjlUU8)

Estimating the Intra-Rater Reliability of Essay Raters

The intra-rater reliability in rating essays is usually indexed by the inter-rater correlation. We suggest an alternative method for estimating intra-rater reliability, in the framework of classical test theory, by using the dis-attenuation formula for inter-test correlations. The validity of the method is demonstrated by extensive simulations, and by applying it to an empirical dataset. It is recommended to use this estimation method whenever the emphasis is not on the average intra-reliability of a group of raters, but when the intra-rater reliability of a specific rater is of interest, e.g. when the error-variance component of the scores is of interest in order to estimate true scores.

Reliability and Structure of the TALIS Social Desirability Scale: An Assessment Based on Item Response Theory

One of the ways of controlling for the influence of social expectations on the answers given by survey respondents is to use a social desirability scale together with the main questions. The social desirability scale, which was included in the Teaching and Learning International Survey (TALIS) international comparative study for this purpose, was used on a Russian-language sample of teachers without cross-cultural adaptation. In addition, this tool was based on the Marlowe-Crowne Social Desirability Scale, whose psychometric characteristics have only been evaluated so far within the framework of classical test theory with mixed results. In order to fill the gap in our understanding of the validity of the TALIS social desirability scale within the framework of item response theory, we analyzed the data obtained from a representative sample of Russian teachers. The results showed that the scale had acceptable reliability, significant unidimensionality, and, at the same time, a number of serious problems with its functionality. We propose measures to improve the quality of the psychometric properties of the scale on the basis of the obtained results, including simulated data. We draw fundamental conclusions about the structure of the social desirability construct.

Power and Sample Size Calculations in Clinical Trials with Patient-Reported Outcomes under Equal and Unequal Group Sizes Based on Graded Response Model: A Simulation Study

ObjectivesTo provide a valid sample size strategy based on simulation and to evaluate the statistical power in clinical trials with patient-reported outcomes (PROs) based on a polytomous item response theory model—the graded response model (GRM)—and to compare this framework with the classical test theory (CTT) approach. MethodsOne thousand randomized clinical trials were simulated using PRO based on the GRM and under various combinations of the number of patients in each arm, the group allocation ratio, the number of items and categories, and group effects. The power and sample size estimated in the simulations were then compared with those computed using the CTT framework. ResultsThe results indicated that the impact of the most influential factors, including the number of patients, group allocation ratio, group effects, and the number of categories, on the power and sample size of the GRM-based and CTT-based approaches was similar. Nevertheless, the strong impact of the number of items on these issues distinguished the two approaches. ConclusionsIt is crucial to use an adapted sample size formula in a GRM-based analysis because the classical formula designed for the CTT-based approach does not consider the impact of the number of items, which could result in an inadequately sized study and a decrease in power. Thus, when clinicians design a randomized clinical trial with polytomous PRO endpoints using classical sample size formula as the base, they should be aware of the possibility of making an incorrect clinical decision.

The Person Centered approach in Gerontology: New validity evidence of the Staff Assessment Person-directed Care Questionnaire

AbstractBackground/Objective Person centered care is a novel approach which aims to improve care quality in services for older people requiring care. The aim of this work is the adaptation and validation of the Staff Assessment Person-directed Care (PDC) in a Spanish population. Method The PDC was applied to a sample of 1,339 direct care professionals from 56 elderly care homes. The psychometric properties were analyzed within the framework of Classical Test Theory and Item Response Theory models. Results The measure showed a high reliability provided by Cronbach's alpha (α = .96), the test-retest reliability (r = .88), and also an adequate Information Function (highest scores between theta values -2 and +2). The factorial structure of PDC is essentially unidimensional, and confirms the existence of two large dimensions which are in turn expressed in eight highly correlated factors. Especially notable in terms of validity evidence based on relations to others variables are the correlations of PDC with the The Person-centered Care Assessment Tool (r= .68), organizational climate (r = .67), emotional exhaustion (r= -.41) and personal accomplishment (r = .45). Conclusions The Spanish version of the PDC demonstrates adequate psychometric properties for its use in the evaluation of elderly care facilities, both for professional and research purposes.

Reflexiones sobre adivinar en preguntas de opción múltiple y cómo afecta el resultado del examen

Multiple-choice tests (MCT) are the most employed assessment tool in medical education; however, its use is limited to the instrument reliability and validity of the inferences made upon the measurement. Guessing, an inherent element of this evaluating tool, may introduce construct-irrelevant variance and reduce the load of the latent trait in the score of the exam due to individual differences regarding educated guessing, testwiseness and guessing tendency. This article presents an overview of Classical Test Theory (CTT) and Item Response Theory (IRT), including a discussion of how each theory addresses the guessing phenomenon. With respect to the latter, we propose a theoretical model that integrates factors related to guessing within the CTT framework. We further include a theoretical analysis, which displays the variation of the probability of passing a MCT reliant on certain assumptions regarding guessing that are akin to a particular IRT model. In conclusion, various features of the items increase the likelihood of guessing, and, when guessing takes place, the magnitude of its effect is determined by some variables that can be dependent or independent of the latent trait.

Measuring Elementary Student’s Mathematics Motivation: A Validity Study

The psychometric properties of a 10-item math motivation scale were empirically validated with an independent sample consisting of 182 elementary-school students. Analysis of the model dimensionality supported a one-factor structure fit. Item parameter estimates from a Classical Test Theory framework revealed that most items were highly discriminating, and the survey is informative for students of low to average math motivation. Differential item functioning (DIF) analyses found two items exhibiting gender bias. Overall, the instrument was a psychometrically valid instrument for measuring math motivation at the elementary level.

A processing speed test based on rule-based item generation: An analysis with the Rasch Poisson Counts model

Processing speed capacity refers to the ability to solve easy tasks in a restricted amount of time. The Münster Mental Speed Test (MS2T) employs the technique of rule-based item generation, which allows to easily produce a large amount of individually different items which nevertheless share the same psychometric properties. For a refined measurement of processing speed, the MS2T comprises items covering several content facets (symbolic, numeric, verbal) as well as varying complexity levels. Processing speed test data is often handled in the classical test theory framework, but in this study we fit the Rasch Poisson Counts model (RPCM), an item response theory (IRT) model especially designed for count data. The RPCM shows a good fit to the data and offers further advantages like a refined item analysis, Rasch scalability, and the possibility to employ more elaborate IRT-based techniques like, for example, computerized adaptive testing.

A DIF detection procedure in multidimensional item response theory framework using MCMC technique

DIF detection procedures are available not only in the classical test theory framework, but also in the IRT-based framework, such as Lord's (1980) chi-square method, Raju's (1988) area measures, and the likelihood ratio test. These methods are very useful in detecting DIF, however, little progress has been made in understanding the causes of DIF. Benign DIF caused by auxiliary dimensions enhances construct validity of a test, while adverse DIF resulting from nuisance dimensions lowers construct validity. Benign DIF items cannot be detected by conducting an additional DIF analysis in which all construct-relevant dimensions are modelled and included in the conditioning variable. Adverse DIF, however, can be eliminated only by deleting the item or by revising it. Therefore, it is very important to have a procedure for investigating the cause of DIF and detecting adverse DIF only. In this study, we developed a DIF detection procedure in multidimensional item response theory framework using MCMC to flag only those items that have adverse DIF. Items of benign DIF detected by other procedures will not be flagged. The DIF detection procedure proposed in this study was applied to both real data and simulated data and was found to be successful.

Relationships Among Classical Test Theory and Item Response Theory Frameworks via Factor Analytic Models.

There are well-defined theoretical differences between the classical test theory (CTT) and item response theory (IRT) frameworks. It is understood that in the CTT framework, person and item statistics are test- and sample-dependent. This is not the perception with IRT. For this reason, the IRT framework is considered to be theoretically superior to the CTT framework for the purpose of estimating person and item parameters. In previous simulation studies, IRT models were used both as generating and as fitting models. Hence, results favoring the IRT framework could be attributed to IRT being the data-generation framework. Moreover, previous studies only considered the traditional CTT framework for the comparison, yet there is considerable literature suggesting that it may be more appropriate to use CTT statistics based on an underlying normal variable (UNV) assumption. The current study relates the class of CTT-based models with the UNV assumption to that of IRT, using confirmatory factor analysis to delineate the connections. A small Monte Carlo study was carried out to assess the comparability between the item and person statistics obtained from the frameworks of IRT and CTT with UNV assumption. Results show the frameworks of IRT and CTT with UNV assumption to be quite comparable, with neither framework showing an advantage over the other.

Taking the Error Term of the Factor Model Into Account

The problem of factor score indeterminacy implies that the factor and the error scores cannot be completely disentangled in the factor model. It is therefore proposed to compute Harman’s factor score predictor that contains an additive combination of factor and error variance. This additive combination is discussed in the framework of classical test theory. On this basis, a definition of reliability, standard error of measurement, and confidence intervals for the factor score predictor are proposed. It is argued that factor score predictor intervals should be used instead of single score predictors to account for the error term in the factor model. The calculation of reliabilities and factor score predictor intervals is illustrated by means of a small simulation study and an empirical example.

Rasch Analysis of the Ocular Surface Disease Index (OSDI)

The Ocular Surface Disease Index (OSDI) is a 12-item scale for the assessment of symptoms related to dry eye disease and their effect on vision. Its reliability and validity have been investigated within the classical test theory framework and, more recently, using Rasch analysis. The purpose of the present analysis was to more completely investigate the functioning of its response category structure, the validity of its three subscales, and the unidimensionality of the latent construct it is intended to assess. Responses to the OSDI from 172 females participating in the Dry Eye in Postmenopause (DEiM) study who had previously been diagnosed with dry eye or reported significant ocular irritation and dryness were analyzed. Response category structure and item fit statistics were evaluated for assessment of model fit. Person separation statistics were used to examine the validity of the subscales. Unidimensionality was assessed by principal component analysis (PCA) of model residuals. The recommended five-category response structure resulted in disordered response thresholds. A four-category structure resulted in ordered thresholds. Item infit statistics were acceptable for all 12 items. Person separation with this category structure was adequate, with a person separation index of 2.16. None of the three subscales demonstrated adequate person separation. PCA showed one other significant factor onto which the three environmental items loaded significantly. All items demonstrated acceptable fit to the model after collapsing categories to order the response thresholds. The original subscales did not prove valid, and there is some evidence of multidimensionality and poor targeting.