Item Response Curves Research Articles

Replicating analyses of item response curves using data from the Force and Motion Conceptual Evaluation

Ishimoto, Davenport, and Wittmann have previously reported analyses of data from student responses to the Force and Motion Conceptual Evaluation (FMCE), in which they used item response curves (IRCs) to make claims about American and Japanese students' relative likelihood to choose certain incorrect responses to some questions. We have used an independent data set of over 6,500 American students' responses to the FMCE to generate IRCs to test their claims. Converting the IRCs to vectors, we used dot product analysis to compare each response item quantitatively. For most questions, our analyses are consistent with Ishimoto, Davenport, and Wittmann, with some results suggesting more minor differences between American and Japanese students than previously reported. We also highlight the pedagogical advantages of using IRCs to determine the differences in response patterns for different populations to better understand student thinking prior to instruction.

Validity of Force Concept Inventory evaluated by students’ explanations and confirmation using modified item response curve

Question 29 of the Force Concept Inventory requires an understanding of air pressure and may not be a good measure of student understanding of forces in certain contexts.

Examining the Dimensionality and Monotonicity of an Attitude Dataset based on the Item Response Theory Models

In the current study, the factor structure of an attitude scale was analyzed by using the two different item response theory models that allow modeling non-monotonic item response curves. The current study utilized the two models to examine whether the two-factor solution of factor analysis may be caused by method effect, or by the failure of the analysis in describing and fitting the dataset because of the monotonicity assumption. This study was conducted on a dataset obtained from 355 undergraduate students who were studying at the Middle East Technical University. The data were obtained by carrying out the Attitude Scale Towards Foreign Languages as Medium of Instruction, which was developed by Kartal and Gülleroğlu (2015). The fit of the scale items to the generalized graded unfolding model was examined based on the item response curves, item parameters, item fit statistics and fit graphics. For Mokken scaling, scalability coefficients were calculated, dimensionality analyzes were conducted by using the Automated Item Selection Procedure. The monotonicity assumption was investigated based on the rest-score group methods. The results of the current study revealed that items of the attitude scale fit to the unidimensional models that do not assume monotone increasing item response curves for all items, while the factor analysis suggested a two-factor solution for the data. Researchers are recommended to utilize statistical techniques that can identify any possible violation of the monotonicity assumption and model items having non-monotonic response curves to examine dimensionality of their data.

More flexible response functions for the PROMIS physical functioning item bank by application of a monotonic polynomial approach.

In developing item banks for patient reported outcomes (PROs), nonparametric techniques are often used for investigating empirical item response curves, whereas final banks usually use parsimonious parametric models. A flexible approach based on monotonic polynomials (MP) provides a compromise by modeling items with both complex and simpler response curves. This paper investigates the suitability of MPs to PRO data. Using PROMIS Wave 1 data (N = 15,725) for Physical Function, we fitted an MP model and the graded response model (GRM). We compared both models in terms of overall model fit, latent trait estimates, and item/test information. We quantified possible GRM item misfit using approaches that compute discrepancies with the MP. Through simulations, we investigated the ability of the MP to perform well versus the GRM under identical data collection conditions. A likelihood ratio test (p < 0.001) and AIC (but not BIC) indicated better fit for the MP. Latent trait estimates and expected test scores were comparable between models, but we observed higher information for the MP in the lower range of physical functioning. Many items were flagged as possibly misfitting and simulations supported the performance of the MP. Yet discrepancies between the MP and GRM were small. The MP approach allows inclusion of items with complex response curves into PRO item banks. Information for the physical functioning item bank may be greater than originally thought for low levels of physical functioning. This may translate into small improvements if an MP approach is used.

Probing for Bias: Comparing Populations Using Item Response Curves

We introduce an approach for making a quantitative comparison of the item response curves (IRCs) of any two populations on a multiple-choice test instrument. In this study, we employ simulated and actual data. We apply our approach to a dataset of 12,187 participants on the 25-item Science Literacy Concept Inventory (SLCI), which includes ample demographic data of the participants. Prior comparisons of the IRCs of different populations addressed only two populations and were made by visual inspection. Our approach allows for quickly comparing the IRCs for many pairs of populations to identify those items where substantial differences exist. For each item, we compute the IRC dot product, a number between 0 and 1 for which a value of 1 occurs when the IRCs of the two populations are identical. We then determine whether the value of the IRC dot product is indicative of significant differences in populations of real students. Through this process, we can quickly discover bias across demographic groups. As a case example, we apply our metric to illuminate four SLCI items that exhibit gender bias. We further found that gender bias was present for non-science majors on those items but not for science majors.

Prioritization of the Skills to Be Mastered for the Daily Jobs of Japanese Dental Hygienists.

Dental hygienists require proficiency in a wide variety of job skills. Dental hygienists should master their job skills step by step, and the prioritization of these steps is important. In this study, we investigated the frequency at which Japanese dental hygienists performed daily jobs and attempted to classify the jobs according to the proficiency levels. The aim of this study was to surmise the order in which skills should be mastered in terms of priority and to investigate the relationship between daily jobs and the motivation for completing jobs. The Japan Dental Hygienists' Association conducts a survey on the employment status of dental hygienists in Japan every five years. The questionnaire is distributed to all members of the Japan Dental Hygienists' Association. In this study, the responses of 3,807 dental hygienists who worked at dental clinics were analyzed. We analyzed 77 kinds of daily jobs and the items regarding the motivation to work. For the analysis, item response theory (IRT), structural equation modeling (SEM), and logistic regression analysis were applied. According to the item response curve, the jobs were classified into 11 clusters. The jobs classified into Cluster 1 were the jobs that most of the average-proficiency Japanese dental hygienists performed. Scaling and root planing were the representative jobs in Cluster 1. Performing the jobs classified into Cluster 5 clearly discriminated whether the dental hygienists were performing multiple jobs. Jobs concerning care for elderly or disabled patients were classified into Cluster 5. Jobs concerning gerodontology, implants, management of staff, and consultations were significantly associated with the motivation to perform jobs. Polishing and adjustment of orthodontic apparatuses was negatively associated with the motivation to perform jobs. Understanding the features of each daily job of dental hygienists is important for planning dental hygienists' lifelong educational programs and evaluating their skill levels and proficiency levels. The results presented in this study may help to reveal the characteristics of dental hygienists' daily jobs.

Item response curve analysis of Likert scale on learning attitudes towards physics

The item response curve (IRC) shows the proportion of item responses plotted against total scores achieved in an assessment instrument. It has acquired widespread employment in the field of physics education to analyze tests on physics content knowledge. In the current study, the IRC was lengthened to investigate attitude items of Colorado Learning Attitudes about Science Survey (CLASS) with a five-point Likert rating format. How well the IRC explained polytomous data and its advantages were considered. Collected data were from 1069 engineering undergraduates at a Thai university. The study focused on 16-item CLASS in component 1 pertaining to personal application and relation to the real world, which conformed to item response theory (IRT) assumptions. It revealed robust positive linear relationships of step parameters and difficulty indices derived between IRC and Rasch analysis, as well as classical test theory. The IRC, a non-modeling method of IRT, exposed not only the item parameters but also features of responses to each category using total scores. It will be helpful in the analysis of both dichotomous and polytomous data.

Exploring the preinstruction and postinstruction non-Newtonian world views as measured by the Force Concept Inventory

Implementing targeted instructional techniques requires an understanding of the initial alternative world views students possess. These alternative world views can be measured using validated conceptual inventories combined with factor analysis methodologies. Using exploratory factor analysis and network module analysis, Scott and Schumayer and Brewe, Bruun, and Bearden identified the most prevalent and “rigid” of these world views which remained in the postinstruction student responses to the Force Concept Inventory. This paper seeks to identify the alternative world views that exist for students before and after instruction through the use of factor analysis applied to pre- and postinstruction responses from 19 727 matched, student responses. Using the distractor taxonomy proposed by Hestenes, Wells, and Swackhamer, the prevalent alternative world views of these students are characterized and analyzed. We find that the pre-instruction and postinstruction coherent alternative world views are similar, suggesting they are learned by the students prior to instruction. Using a modified version of item response curves, it was found that the active force misconception is the most rigid of the misconceptions found pre- and postinstruction, and shows minor growth in these response rates at the end of instruction. The results of this study are consistent with what experienced instructors observe when determining what physics preconceptions incoming students have. This study’s agreement, combined with the results of Scott and Schumayer and Brewe, Bruun, and Bearden, strengthens the physics education research community’s suggestions for improving physics instruction when using only anecdotal or qualitative evidence.10 MoreReceived 20 December 2018DOI:https://doi.org/10.1103/PhysRevPhysEducRes.15.010123Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAssessmentPhysics Education Research

PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition.

By providing substantial amounts of data and standardized evaluation protocols, datasets in computer vision have helped fuel advances across all areas of visual recognition. But even in light of breakthrough results on recent benchmarks, it is still fair to ask if our recognition algorithms are doing as well as we think they are. The vision sciences at large make use of a very different evaluation regime known as Visual Psychophysics to study visual perception. Psychophysics is the quantitative examination of the relationships between controlled stimuli and the behavioral responses they elicit in experimental test subjects. Instead of using summary statistics to gauge performance, psychophysics directs us to construct item-response curves made up of individual stimulus responses to find perceptual thresholds, thus allowing one to identify the exact point at which a subject can no longer reliably recognize the stimulus class. In this article, we introduce a comprehensive evaluation framework for visual recognition models that is underpinned by this methodology. Over millions of procedurally rendered 3D scenes and 2D images, we compare the performance of well-known convolutional neural networks. Our results bring into question recent claims of human-like performance, and provide a path forward for correcting newly surfaced algorithmic deficiencies.

A 67-item stress resilience item bank showing high content validity was developed in a psychosomatic sample

ObjectivesTo develop the first item bank to measure stress resilience (SR) in clinical populations. Study Design and SettingQualitative item development resulted in an initial pool of 131 items covering a broad theoretical SR concept. These items were tested in n = 521 patients at a psychosomatic outpatient clinic. Exploratory and confirmatory factor analysis, as well as other state-of-the-art item analyses and item response theory were used for item evaluation and calibration of the final item bank. ResultsOf the initial item pool of 131 items, we excluded 64 items (54 factor loading <0.5, four residual correlations >0.3, two nondiscriminative item response curves, and four differential item functioning). The final set of 67 items indicated sufficient model fit in confirmatory factor analysis and item response theory analyses. In addition, a 10-item short form with high measurement precision (SE ≤ 0.32 in a theta range between −1.8 and +1.5) was derived. Both the SR item bank and the SR short form were highly correlated with an existing static legacy tool (Connor-Davidson Resilience Scale). ConclusionThe final SR item bank and 10-item short form showed good psychometric properties. When further validated, they will be ready to be used within a framework of computer-adaptive tests for a comprehensive assessment of the stress construct.

Use of item response curves of the Force and Motion Conceptual Evaluation to compare Japanese and American students’ views on force and motion

Item response curves can be used as part of the validation process when extending multiple-choice instruments to new contexts, such as languages and nationalities.

Relapse Risk Assessment for Schizophrenia Patients (RASP): A New Self-Report Screening Tool.

The Relapse Assessment for Schizophrenia Patients (RASP) was developed as a six-question self-report screener that measures indicators of Increased Anxiety and Social Isolation to assess patient stability and predict imminent relapse. This paper describes the development and psychometric characteristics of the RASP. The RASP and Positive and Negative Syndrome Scale (PANSS) were administered to patients with schizophrenia (n=166) three separate times. Chart data were collected on a subsample of patients (n=81). Psychometric analyses of RASP included tests of reliability, construct validity, and concurrent validity of items. Factors from RASP were correlated with subscales from PANSS (sensitivity to change and criterion validity [agreement between RASP and evidence of relapse]). Test-retest reliability returned modest to strong agreement at the item level and strong agreement at the questionnaire level. RASP showed good item response curves and internal consistency for the total instrument and within each of the two subscales (Increased Anxiety and Social Isolation). RASP Total Score and subscales showed good concurrent validity when correlated with PANSS Total Score, Positive, Excitement, and Anxiety subscales. RASP correctly predicted relapse in 67% of cases, with good specificity and negative predictive power and acceptable positive predictive power and sensitivity. The reliability and validity data presented support the use of RASP in settings where addition of a brief self-report assessment of relapse risk among patients with schizophrenia may be of benefit. Ease of use and scoring, and the ability to administer without clinical supervision allows for routine administration and assessment of relapse risk.

Analysis test of understanding of vectors with the three-parameter logistic model of item response theory and item response curves technique

An example of the use of item response theory to assess multiple-choice conceptual instruments.

Development and validation of a learning progression for change of seasons, solar and lunar eclipses, and moon phases

In this paper, we report about the development and validation of a learning progression about the Celestial Motion big idea. Existing curricula, research studies on alternative conceptions about these phenomena, and students’ answers to an open questionnaire were the starting point to develop initial learning progressions about change of seasons, solar and lunar eclipses, and Moon phases; then, a two-tier multiple choice questionnaire was designed to validate and improve them. The questionnaire was submitted to about 300 secondary students of different school levels (14 to 18 years old). Item response analysis and curve integral method were used to revise the hypothesized learning progressions. Findings support that spatial reasoning is a key cognitive factor for building an explanatory framework for the Celestial Motion big idea, but also suggest that causal reasoning based on physics mechanisms underlying the phenomena, as light flux laws or energy transfers, may significantly impact a students’ understanding. As an implication of the study, we propose that the teaching of the three discussed astronomy phenomena should follow a single teaching-learning path along the following sequence: (i) emphasize from the beginning the geometrical aspects of the Sun-Moon-Earth system motion; (ii) clarify consequences of the motion of the Sun-Moon-Earth system, as the changing solar radiation flow on the surface of Earth during the revolution around the Sun; (iii) help students moving between different reference systems (Earth and space observer’s perspective) to understand how Earth’s rotation and revolution can change the appearance of the Sun and Moon. Instructional and methodological implications are also briefly discussed.9 MoreReceived 27 September 2014DOI:https://doi.org/10.1103/PhysRevSTPER.11.020102This article is available under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical Society

A Quasi-Parametric Method for Fitting Flexible Item Response Functions

If standard two-parameter item response functions are employed in the analysis of a test with some newly constructed items, it can be expected that, for some items, the item response function (IRF) will not fit the data well. This lack of fit can also occur when standard IRFs are fitted to personality or psychopathology items. When investigating reasons for misfit, it is helpful to compare item response curves (IRCs) visually to detect outlier items. This is only feasible if the IRF employed is sufficiently flexible to display deviations in shape from the norm. A quasi-parametric IRF that can be made arbitrarily flexible by increasing the number of parameters is proposed for this purpose. To take capitalization on chance into account, the use of Akaike information criterion or Bayesian information criterion goodness of approximation measures is recommended for suggesting the number of parameters to be retained. These measures balance the effect on fit of random error of estimation against systematic error of approximation. Computational aspects are considered and efficacy of the methodology developed is demonstrated.

Is the adapted Household Food Insecurity Access Scale (HFIAS) developed internationally to measure food insecurity valid in urban and rural households of Ethiopia?

The concept of food insecurity encompasses three dimensions. One of these dimensions, the access component of household food insecurity is measured through the use of the Household Food Insecurity Access Scale (HFIAS). Despite its application in Ethiopia and other similar developing countries, its performance is still poorly explored. Our study aims to evaluate the validity of the HFIAS in Ethiopia. We conducted repeated cross-sectional studies in urban and rural villages of the Butajera District in southern Ethiopia. The validation was conducted on a pooled sample of 1,516 households, which were selected using a simple random sampling method. The HFIAS was translated into the local Amharic language and tested for face validity. We also evaluated the tool’s internal consistency using Cronbach’s alpha and factor analysis. We tested for parallelism on HFIAS item response curves across wealth status and further evaluated the presence of a dose-response relationship between the food insecurity level and the consumption of food items, as well as between household wealth status and food insecurity. Additionally, we evaluated the reproducibility of the tool through the first and second round of HFIAS scores. The HFIAS exhibited a good internal consistency (Cronbach’s alpha for the values of rounds 1 and 2 were 0.76 and 0.73, respectively). A factor analysis (varimax rotation) resulted in two main factors: the first factor described a level of mild to moderate food insecurity, while the second factor described severe food insecurity. HFIAS item response curves were parallel across wealth status in the sample households, with a dose-response trend between food insecurity levels and the likelihood of previous day food consumption being observed. The overall HFIAS score did not change over the two rounds of data collection. The HFIAS is a simple and valid tool to measure the access component of household food insecurity. However, we recommend the adaptation of questions and wordings and adding examples before application, as we found a discrepancy in understanding of some of the nine HFIAS questions.

A new computerized adaptive test advancing the measurement of health-related quality of life (HRQoL) in children: the Kids-CAT.

Assessing health-related quality of life (HRQoL) via Computerized Adaptive Tests (CAT) provides greater measurement precision coupled with a lower test burden compared to conventional tests. Currently, there are no European pediatric HRQoL CATs available. This manuscript aims at describing the development of a HRQoL CAT for children and adolescents: the Kids-CAT, which was developed based on the established KIDSCREEN-27 HRQoL domain structure. The Kids-CAT was developed combining classical test theory and item response theory methods and using large archival data of European KIDSCREEN norm studies (n = 10,577-19,580). Methods were applied in line with the US PROMIS project. Item bank development included the investigation of unidimensionality, local independence, exploration of Differential Item Functioning (DIF), evaluation of Item Response Curves (IRCs), estimation and norming of item parameters as well as first CAT simulations. The Kids-CAT was successfully built covering five item banks (with 26-46 items each) to measure physical well-being, psychological well-being, parent relations, social support and peers, and school well-being. The Kids-CAT item banks proved excellent psychometric properties: high content validity, unidimensionality, local independence, low DIF, and model conform IRCs. In CAT simulations, seven items were needed to achieve a measurement precision between .8 and .9 (reliability). It has a child-friendly design, is easy accessible online and gives immediate feedback reports of scores. The Kids-CAT has the potential to advance pediatric HRQoL measurement by making it less burdensome and enhancing the patient-doctor communication.

Enhancing the Equating of Item Difficulty Metrics: Estimation of Reference Distribution

Two methods are currently in use at Educational Testing Service (ETS) for equating observed item difficulty statistics. The first method involves the linear equating of item statistics in an observed sample to reference statistics on the same items. The second method, or the item response curve (IRC) method, involves the summation of conditional observed item statistics across the reference population total score frequencies. This article introduces a quick and effective method for obtaining the reference distribution for the transition from the linear equating method to the IRC method without recalculating all the item difficulties. More specifically, a mathematical formula is derived to estimate the score distribution of a reference group that maintains the current item difficulty scale. Future research is needed to compare the performance of the two approaches.

Investigating invariant item ordering in the Mental Health Inventory: An illustration of the use of different methods

Invariant item ordering is a property of scales whereby the items are scored in the same order across a wide range of the latent trait and across a wide range of respondents. In the package ‘mokken’ in the statistical software R, the ability to analyse Mokken scales for invariant item ordering has recently been available and techniques for inspecting visually the item response curves of item pairs, have also been included. While methods to assess invariant item ordering are available, there have been indications that items representing extremes of distress in mental well-being scales, such as suicidal ideation, may lead to claiming invariant item ordering where it does not exist. We used the Mental Health Inventory to see if invariant item ordering was indicated in any Mokken scales derived and to see if this was being influenced by extreme items. A Mokken scale was derived indicating invariant item ordering. Visual inspection of the item pairs indicated that the most difficult item (suicidal ideation) was located far from the remaining cluster of items. Removing this item lowered invariant item ordering to an unacceptable level.

Erratum: “An item response curves analysis of the force concept inventory” [Am. J. Phys. 80, 825–831 (2012)

Erratum: “An item response curves analysis of the force concept inventory” [Am. J. Phys. 80, 825–831 (2012)