Item Characteristic Curves Research Articles

A Strategy for Replacing Sum Scoring

This article promotes the use of modern test theory in testing situations where sum scores for binary responses are now used. It directly compares the efficiencies and biases of classical and modern test analyses and finds an improvement in the root mean squared error of ability estimates of about 5% for two designed multiple-choice tests and about 12% for a classroom test. A new parametric density function for ability estimates, the tilted scaled β, is used to resolve the nonidentifiability of the univariate test theory model. Item characteristic curves (ICCs) are represented as basis function expansions of their log-odds transforms. A parameter cascading method along with roughness penalties is used to estimate the corresponding log odds of the ICCs and is demonstrated to be sufficiently computationally efficient that it can support the analysis of large data sets.

A Bayesian Robust IRT Outlier-Detection Model.

In psychometric practice, the parameter estimates of a standard item-response theory (IRT) model can become biased when item-response data, of persons' individual responses to test items, contain outliers relative to the model. Also, the manual removal of outliers can be a time-consuming and difficult task. Besides, removing outliers leads to data information loss in parameter estimation. To address these concerns, a Bayesian IRT model that includes person and latent item-response outlier parameters, in addition to person ability and item parameters, is proposed and illustrated, and is defined by item characteristic curves (ICCs) that are each specified by a robust, Student's t-distribution function. The outlier parameters and the robust ICCs enable the model to automatically identify item-response outliers, and to make estimates of the person ability and item parameters more robust to outliers. Hence, under this IRT model, it is unnecessary to remove outliers from the data analysis. Our IRT model is illustrated through the analysis of two data sets, involving dichotomous- and polytomous-response items, respectively.

An analysis of the mathematics school examination test quality

This research aims to describe: (1) the quality of mathematics school examination test of the Junior High School for the academic year 2015/2016 in Kabupaten Bangkalan based on qualitative analysis of tests item, (2) ) the quality of mathematics school examination test of Junior High School for the academic year 2015/2016 in Kabupaten Bangkalan based on quantitative analysis of test items, and (3) the test equating on mathematics school examination test for the academic year of 2015/2016 of Junior High School in Kabupaten Bangkalan. A test is said to be quality if it fulfills validity, reliability, and it has good characteristic. A test is said to be equivalent to another if the test scores of one test can be exchanged with the scores of another test. The data is taken from school examination script accompanied with the answer sheets of students. Qualitative data analysis was conducted with the help of experts judgement. Quantitative data analysis was conducted with Classical Test Theory by Iteman and Item Response Theory by BilogMG. These programs are used to find out the test quality quantitatively. In order to analyze the equivalence between series of tests, item-characteristic curves were used. These curves were drawn by Geogebra. The research result shows: (1) qualitatively, the test plan quality of mathematics school examination test is quite good. Meanwhile, the school exam quality is quite good and not so good, (2) quantitatively, the school examination test quality is good, and (3) equating, based on the item-characteristic curves, the school examination tests is equal.

Linking Methods for the Zinnes-Griggs Pairwise Preference IRT Model.

Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have increased, there has been a growing need for methods to link parameters estimated in different examinee groups as a prelude to measurement equivalence testing. This study compared four linking methods for the Zinnes and Griggs (ZG) pairwise preference ideal point model. A Monte Carlo simulation compared test characteristic curve (TCC) linking, item characteristic curve (ICC) linking, mean/mean (M/M) linking, and mean/sigma (M/S) linking. The results indicated that ICC linking and the simpler M/M and M/S methods performed better than TCC linking, and there were no substantial differences among the top three approaches. In addition, in the absence of possible contamination of the common (anchor) item subset due to differential item functioning, five items should be adequate for estimating the metric transformation coefficients. Our article presents the necessary equations for ZG linking and provides recommendations for practitioners who may be interested in developing and using pairwise preference measures for research and selection purposes.

IRT Item Parameter Scaling for Developing New Item Pools

ABSTRACTIncreasing use of item pools in large-scale educational assessments calls for an appropriate scaling procedure to achieve a common metric among field-tested items. The present study examines scaling procedures for developing a new item pool under a spiraled block linking design. The three scaling procedures are considered: (a) concurrent calibration, (b) separate calibration with one linking, and (c) separate calibration with three sequential linking. Evaluation across varying sample sizes and item pool sizes suggests that calibrating an item pool simultaneously results in the most stable scaling. The separate calibration with linking procedures produced larger scaling errors as the number of linking steps increased. The Haebara’s item characteristic curve linking resulted in better performances than the test characteristic curve (TCC) linking method. The present article provides an analytic illustration that the test characteristic curve method may fail to find global solutions in polytomous items. Finally, comparison of the single- and mixed-format item pools suggests that the use of polytomous items as the anchor can improve the overall scaling accuracy of the item pools.

EHealth literacy in chronic disease patients: An item response theory analysis of the eHealth literacy scale (eHEALS)

BackgroundChronic disease patients are affected by low computer and health literacy, which negatively affects their ability to benefit from access to online health information. ObjectiveTo estimate reliability and confirm model specifications for eHealth Literacy Scale (eHEALS) scores among chronic disease patients using Classical Test (CTT) and Item Response Theory techniques. MethodsA stratified sample of Black/African American (N=341) and Caucasian (N=343) adults with chronic disease completed an online survey including the eHEALS. Item discrimination was explored using bi-variate correlations and Cronbach’s alpha for internal consistency. A categorical confirmatory factor analysis tested a one-factor structure of eHEALS scores. Item characteristic curves, in-fit/outfit statistics, omega coefficient, and item reliability and separation estimates were computed. ResultsA 1-factor structure of eHEALS was confirmed by statistically significant standardized item loadings, acceptable model fit indices (CFI/TLI>0.90), and 70% variance explained by the model. Item response categories increased with higher theta levels, and there was evidence of acceptable reliability (ω=0.94; item reliability=89; item separation=8.54). ConclusioneHEALS scores are a valid and reliable measure of self-reported eHealth literacy among Internet-using chronic disease patients. Practice implicationsProviders can use eHEALS to help identify patients’ eHealth literacy skills.

Conceptualizing and Measuring Weekend versus Weekday Alcohol Use: Item Response Theory and Confirmatory Factor Analysis.

Culturally, people tend to abstain from alcohol intake during the weekdays and wait to consume in greater frequency and quantity during the weekends. The current research sought to empirically justify the days representing weekday versus weekend alcohol consumption. In study 1 (N = 419), item response theory was applied to a two-parameter (difficulty and discrimination) model that evaluated the days of drinking (frequency) during the typical 7-dayweek. Item characteristic curves were most similar for Monday, Tuesday, and Wednesday (prototypical weekday) and for Friday and Saturday (prototypical weekend). Thursday and Sunday, however, exhibited item characteristics that bordered the properties of weekday and weekend consumption. In study 2 (N = 403), confirmatory factor analysis was applied to test six hypothesized measurement structures representing drinks per day (quantity) during the typical week. The measurement model producing the strongest fit indices was a correlated two-factor structure involving separate weekday and weekend factors that permitted Thursday and Sunday to double load on both dimensions. The proper conceptualization and accurate measurement of the days demarcating the normative boundaries of "dry" weekdays and "wet" weekends are imperative to inform research and prevention efforts targeting temporal alcohol intake patterns.

A comparison of four differential Item functioning procedures in the presence of multidimensionality

Differential item functioning (DIF), or item bias, is a relatively new concept. It has been one of the most controversial and the most studied subject in measurement theory. DIF occurs when people who have the same ability level but from different groups have a different probability of a correct response. According to Item Response Theory (IRT), DIF occurs when item characteristic curves (ICC) of two groups are not identical or do not have the same item parameters after rescaling. Also, DIF might occur when latent ability space is misspecified. When the groups have different multidimensional ability distributions and test items chosen to discriminate among these abilities, using unidimensional scoring, might flag items as DIF items. The purpose of this study was to compare four DIF procedures the Mantel Haenszel (MH), the Simultaneous Item Bias Test (SIBTEST), the IRT, the Logistic Regression (LR) when the underlying ability distribution is erroneously assumed to be homogenous. To illustrate the effect of assuming a homogenous ability distribution for the groups while they differ in terms of their underlying multidimensional ability levels on the DIF procedures, two different data sets were generated; one set in which DIF occurs, and one set in which no DIF occurs by using 2PL model. The UNIGEN program was used to generate the data. Each of the data sets contained 1000 examinees and 25 items. Item parameters where chosen to be capable of measuring a two dimensional ability distribution of the two groups. The MH, the SIBTEST, the AREA and the LR procedures were applied to the data both with DIF and without DIF. The study showed that all the four methods identified items as biased when the ability space was misspecified. Key words: Item response theory, simultaneous A±tem bias test (SIBTEST), differential item functioning, differential item functioning (DIF), Mantel Haenszel (MH), logistic regression (LR).

Computer Adaptive Testing and Cloud Computing

Computer Adaptive Testing (CAT) is an example of a Com- puter Based Test (CBT) and is one of the main trending topics in the area of knowledge testing and, more recently, in e{learning or in Intelligent Tutoring Systems scenarios. The Item Response Theory (IRT) denes the theoretical basis of a CAT implementation, which assumes the existence of a repository of properly calibrated items that is used during the testing process of a particular examinee. The calibration and adaptation are based on an Item Characteristic Curve (ICC) related to an specic model, being Rasch's models the most widely used. CAT systems require high computational cost to implement the calibration and evaluation processes and the amount of concurrent users at a time could be large enough. Thus, the platform must support high concurrency and availability to perform a desired level of functionality. Technological tendencies in computing oer each time better platforms to develop and manage big collections of data for its processing and relevant information extraction. This paper presents a perspective of using new technologies in CAT as an alternative of implementation. Particularly, the use of a cloud computing platform as current alternative for online CAT systems using the capabilities of multicore processing and big amount of RAM that oers the cloud, to resolve the proper mathematical equations related to psychometric models and the operations described in their algorithms in a real evaluation scheme.

Effects of Initial Values and Convergence Criterion in the Two-Parameter Logistic Model When Estimating the Latent Distribution in BILOG-MG 3.

Parameters of the two-parameter logistic model are generally estimated via the expectation-maximization algorithm, which improves initial values for all parameters iteratively until convergence is reached. Effects of initial values are rarely discussed in item response theory (IRT), but initial values were recently found to affect item parameters when estimating the latent distribution with full non-parametric maximum likelihood. However, this method is rarely used in practice. Hence, the present study investigated effects of initial values on item parameter bias and on recovery of item characteristic curves in BILOG-MG 3, a widely used IRT software package. Results showed notable effects of initial values on item parameters. For tighter convergence criteria, effects of initial values decreased, but item parameter bias increased, and the recovery of the latent distribution worsened. For practical application, it is advised to use the BILOG default convergence criterion with appropriate initial values when estimating the latent distribution from data.

Examining the Process of Responding to Circumplex Scales of Interpersonal Values Items: Should Ideal Point Scoring Methods Be Considered?

ABSTRACTThe Circumplex Scales of Interpersonal Values (CSIV) is a 64-item self-report measure of goals from each octant of the interpersonal circumplex. We used item response theory methods to compare whether dominance models or ideal point models best described how people respond to CSIV items. Specifically, we fit a polytomous dominance model called the generalized partial credit model and an ideal point model of similar complexity called the generalized graded unfolding model to the responses of 1,893 college students. The results of both graphical comparisons of item characteristic curves and statistical comparisons of model fit suggested that an ideal point model best describes the process of responding to CSIV items. The different models produced different rank orderings of high-scoring respondents, but overall the models did not differ in their prediction of criterion variables (agentic and communal interpersonal traits and implicit motives).

Exploring Crossing Differential Item Functioning by Gender in Mathematics Assessment

The purpose of this article is to explore crossing differential item functioning (DIF) in a test drawn from a national examination of mathematics for 11-year-old pupils in England. An empirical dataset was analyzed to explore DIF by gender in a mathematics assessment. A two-step process involving the logistic regression (LR) procedure for detecting uniform and nonuniform DIF was applied to identify crossing DIF. The results showed 36 uniform and 19 nonuniform statistically significant gender DIF items. Out of the 19 nonuniform DIF items, 10 items were crossing DIF. We explained nonuniform DIF using the crossing point in item characteristic curves and the LR-DIF coding scheme. This study showed that crossing DIF exists in empirical data and the findings from this study provide a potentially valuable contribution in understanding such items.

English

The purpose of this study is to carry out differential item functioning (DIF) analysis for content areas of a reading comprehension subtest using four area indices within Item Response Theory (IRT) framework. The differences in the magnitudes of the area indices were compared based on the subject areas. The DIF analysis was carried out across gender groups only. The item level data of the English reading comprehension subtest were gathered from English Language Achievement exam done in School of Foreign Languages, Ege University, Turkey, in 2013. A sample of 2,117 examinees (1,011 males and 1,116 females) was randomly selected. For the DIF analysis, (a) an IRT model for the item characteristic curves was specified; (b) model-data-fit was investigated for the selected IRT model;(c) Item characteristic curves were separately computed for each group on a common scale; finally, (d) indices indicating the degree of DIF on each item were computed. The results of the study indicated that both un-weighted and weighted area indices showed non-uniformity in DIF in the item characteristic curves in reading comprehension subtest in most cases. A significant correlation was observed between un-weighted and weighted area indices. Key words: Item characteristic curve, item bias, differential item functioning (DIF).

Development and validation of a socioculturally competent trust in physician scale for a developing country setting

Trust in physicians is the unwritten covenant between the patient and the physician that the physician will do what is in the best interest of the patient. This forms the...

A new visualization and conceptualization of categorical longitudinal development: measurement invariance and change.

The Overlapping Waves Model (OWM) is a metaphor introduced by Siegler (1996) to illustrate a typical sequence of increasing and decreasing use of strategies during development. Going beyond metaphor, a new model synthesized from Latent Growth Modeling (LGM) and Item Response Theory (IRT) will be presented to analyze such categorical longitudinal data. Use of strategies can be scored as a variable with only a few ordinal categories. IRT provides the means to relate the usage of strategies to position on an underlying developmental dimension. LGM allows to model movement of individuals along this dimension, acknowledging individual differences both in starting point and in speed of progress. Measuring and modeling such strategy development requires that at each time point the same categories are used, in the sense that item difficulties must remain invariant over time. Whether, discrimination can be relaxed is still an issue. The problem that had to be solved was disentangling the between-person-individual differences from real intra-individual developmental differences. Figures with polytomous or multi-category Item Characteristic Curves (ICC's) resemble the OWM in many respects. However, such figures are usually taken to represent inter-individual differences, whereas the OWM usually represents development (so intra-individual differences), and we cannot have both at the same time. The solution came from creating a framework with ability differences on one axis and the effect of time on another axis, resulting in a 3-D model. These (orthogonal) dimensions make it possible to adequately conceptualize measurement invariance in this complex context. As the result is difficult to conceptualize without extensive visualization, special 3-D figures will be used to illustrate and a dynamic (rotatable and scalable) version will be made available as Computable Document Format object (Mathematica). The model was successfully applied in several microgenetic studies.

Differentiation of Self Inventory—Short Form: Development and Preliminary Validation

In a series of three studies we constructed the Differentiation of Self Inventory—Short Form (DSI-SF) and established initial evidence of reliability and validity. In study 1, we used Item Response Theory to reduce the number of items on the Differentiation of Self Inventory—Revised (Skowron and Schmitt in J Marital Fam Ther 29(2):209–222, 2003. doi: 10.1111/j.1752-0606.2003.tb01201.x ). DSI-SF scale items were chosen based on full length scale item content and item characteristic curves. Study 2 provided evidence for criterion, construct, and structural validity of the DSI-SF. Specifically, scores on the DSI-SF were positively related to self-esteem and negatively related to symptoms of depression, state anxiety, trait anxiety, and perceived stress. Evidence of convergent validity was found in a positive relationship between the subscales of the DSI-SF and the Level of Differentiation of Self Scale (Haber in The measurement of nursing outcomes, 2nd edn. Springer, New York, pp 320–331, 2003). The DSI-SF subscales and full scale were also shown to retain most of the reliability of the full length scales. In the third study, preliminary estimates of 4-week test–retest reliability ranged from 0.72 (Fusion with Others subscale) to 0.85 (DSI-SF Full Scale). These results support the DSI-SF in college student samples.

Developing scales measuring disorder-specific intolerance of uncertainty (DSIU): A new perspective on transdiagnostic

Intolerance of uncertainty (IU) is a construct of growing prominence in literature on anxiety disorders and major depressive disorder. Existing measures of IU do not define the uncertainty that respondents perceive as distressing. To address this limitation, we developed eight scales measuring disorder-specific intolerance of uncertainty (DSIU) relating to various anxiety disorders and major depressive disorder. We used exploratory factor analysis and item characteristic curves in two large undergraduate samples (Ns=627 and 628) to derive eight three-item DSIU scales (24 items total) that exhibited excellent psychometric properties. Confirmatory factor analysis supported the factor structures of the scales and the transdiagnostic nature of IU. Each scale predicted unique variance in its respective symptom measure beyond a traditional measure of IU. DSIU represents a theoretically proximal and causal intermediary between known vulnerability factors and disorder symptomatology. The DSIU scales can be used to advance theories of psychopathology and inform case conceptualization and treatment planning.

Do people with and without medical conditions respond similarly to the Short Health Anxiety Inventory? An assessment of differential item functioning using item response theory

ObjectiveIndividuals with medical conditions are likely to have elevated health anxiety; however, research has not demonstrated how medical status impacts response patterns on health anxiety measures. Measurement bias can undermine the validity of a questionnaire by overestimating or underestimating scores in groups of individuals. We investigated whether the Short Health Anxiety Inventory (SHAI), a widely-used measure of health anxiety, exhibits medical condition-based bias on item and subscale levels, and whether the SHAI subscales adequately assess the health anxiety continuum. MethodsData were from 963 individuals with diabetes, breast cancer, or multiple sclerosis, and 372 healthy individuals. Mantel–Haenszel tests and item characteristic curves were used to classify the severity of item-level differential item functioning in all three medical groups compared to the healthy group. Test characteristic curves were used to assess scale-level differential item functioning and whether the SHAI subscales adequately assess the health anxiety continuum. ResultsNine out of 14 items exhibited differential item functioning. Two items exhibited differential item functioning in all medical groups compared to the healthy group. In both Thought Intrusion and Fear of Illness subscales, differential item functioning was associated with mildly deflated scores in medical groups with very high levels of the latent traits. Fear of Illness items poorly discriminated between individuals with low and very low levels of the latent trait. ConclusionsWhile individuals with medical conditions may respond differentially to some items, clinicians and researchers can confidently use the SHAI with a variety of medical populations without concern of significant bias.

Distance Discrimination Method based on Conditional Expectation in Cognitive Diagnosis

The primary purpose for cognitive diagnostic assessment is to classify examinees into mutually exclusive categories. The current practice of obtaining classification categories relies on the distance between the ideal and observed response patterns, such as generalized distance discrimination method(Sun et al., 2011, 2013) and nonparametric approach(Chiu and Douglas, 2013). In these methods, an appropriate set of ideal response patterns can be computed from the universal set of knowledge states and a Q matrix(Tatsuoka, 1995; Leighton et al., 2004; Ding et al., 2009, 2010). However, the ideal response pattern is generated for each knowledge state without considering the stochastic nature of item response, which is contrary to real test situations. For example, examinees who have mastered some of attributes required to solve a particular item can have a higher probability of answering it correctly than less able examinees having mastered none of the attributes. The purpose of this study is focused on choosing the center for each of the classification categories or clusters that is representative of the data. Since this task probably requires knowledge on the distribution of the data, as a practical matter, it is often reasonable to assume that item response pattern is a random vector with a discrete conditional distribution for each knowledge state. The conditional expected vector is considered to be class-center. Once the center is decided for each knowledge state, observed responses pattern can be assigned to the closest class according to minimum Euclidean distance classifier, well-known for measuring the distance between observed response pattern and the center. This method is called distance discrimination method. The conditional distribution can be defined by item response function(IRF) in cognitive diagnostic models. However, many existing item banks are developed under the framework of item response theory. In this case, we propose a method utilizing the information of the item characteristic curve(ICC) or IRF of item response theory model to estimate the IRF for cognitive diagnostic model. It is based on a nonparametric regression approach to transform the IRF of item response theory model into that of cognitive diagnostic model. The resulting IRF is used to classify examinees using minimum Euclidean distance classifier. To investigate whether this method can work under certain conditions, simulated data were generated with six attributes. Four important factors were considered:(a) the source of the attribute structure(the linear hierarchy, the convergent hierarchy, the divergent hierarchy, the unstructured hierarchy, the independent hierarchy),(b) the number of examinees(N = 300, 500, 1,000),(c) two cognitive diagnostic models(the deterministic inputs, noisy "and" gate model and the reduced reparametrized unified model);(d) the quality of the items(s, g ~ U(0.05, 0.25) or U(0.05, 0.4), ? *~U(0.8, 0.98) and r*~U(0.1, 0.6) or ? *~U(0.75, 0.95) and r*~U(0.2, 0.95)). The results show that the estimation method of IRF is promising in terms of precision, and distance discrimination method based on conditional expectation works well in terms of accuracy, especially when the R-RUM fitted the data with low quality of test items. In addition, the fact that the IRF transform method between cognitive diagnostic model and item response model may contribute immensely to test equating of cognitive diagnostic tests. In discussion, we also explain the relationship between nonparametric approach, generalized distance discrimination method and rule space method.

%lrasch_mml: ASASMacro for Marginal Maximum Likelihood Estimation in Longitudinal Polytomous Rasch Models

Item response theory models are often applied when a number items are used to measure a unidimensional latent variable. Originally proposed and used within educational research, they are also used when focus is on physical functioning or psychological wellbeing. Modern applications often need more general models, typically models for multidimensional latent variables or longitudinal models for repeated measurements. This paper describes a SAS macro that fits two-dimensional polytomous Rasch models using a specification of the model that is sufficiently flexible to accommodate longitudinal Rasch models. The macro estimates item parameters using marginal maximum likelihood estimation. A graphical presentation of item characteristic curves is included.