Item Fit Research Articles

Measuring Faculty Engagement in Online Formative or Whole-Person Education

The COVID-19 pandemic reinforced the importance of supporting students’ comprehensive well-being when teaching online. One promising approach is formative or whole-person education, which emphasizes wholeness, purpose, and community. We created a scale using a polytomous Item Response Theory modeling approach, measuring the extent to which postsecondary teachers engage in formative education online. To our knowledge, this is the first scale designed to measure this construct. The scale was developed within an exploratory sequential mixed methods study on formative education online that also included semi-structured interviews with 37 faculty members. Results from the qualitative analysis were used to develop initial items. This data-informed process increased the construct validity of the scale. We refined the original item pool through a pilot test using a sample of 308 instructors. This article presents psychometric results for the final, 10-item scale using a sample of 245 instructors. Evaluation of item fit statistics, item trace lines, and the total information curve indicate that the graded response model was appropriate for this scale. The Cronbach’s alpha and marginal reliability coefficients for the final scale were .90 and .91, indicating good reliability. Future research can explore how this scale might be adapted for in-person learning environments and other contexts.

Assessing adaptation of the psychometric properties of the General Health Questionnaire (GHQ-12) using the Rasch Measurement Model among Indian teenagers

The General Health Questionnaire (GHQ-12) is widely accepted as an instrument used to examine diagnosable psychiatric disorders. It is also routinely used as a measure of psychological morbidity. This study aimed to assess and establish the psychometric properties of the GHQ-12 among Indian teenagers. The tool was administered to 212 adolescents aged from 12 to 18 years. Following two-stage cluster sampling, the data were collected from Aligarh district, India. Fleiss Kappa analysis was used to determine test reliability and showed an overall value of 0.94, based on the rater agreement for the instrument. Furthermore, the Rasch measurement model was used, with values of 0.79 and 0.83 for person and item reliability, respectively. Moreover, the value of unidimensionality was found to be 37.9%. Additionally, item fit statistics and item analysis were conducted for the instrument. Based on the preliminary data and findings, the study provides primary evidence for the reliability and validity of GHQ-12. Hence, the questionnaire can lead to more multi-site studies in India.

Measurement of Psychometric Properties Numerical Aptitude Assessment Scale for Prospective High School Students: A Rasch Model Analysis

Developing structured test instruments is essential in forming a measurable and consistent ability assessment construct. This study examines the psychometric properties of the numerical aptitude test instrument (NAPTIN) utilizing Rasch model analysis to evaluate standardized numerical aptitude assessment test instruments for prospective high school (PHS) students based on validation and construct reliability. Data was collected using an online survey of 228 PHS students. The Rasch model analyses person and item fit, item measure, dimensionality and local independence testing, reliability, and differential item functioning. The results showed that NAPTIN had appropriate and consistent items for measuring the numerical aptitude of PHS students based on gender and school type. This study recommends a systematic measurement tool for researchers. Another contribution is measurement instruments for Indonesian schools and policymakers to choose student program specialties.

Development of Instrument of Emotional Climate and Attitude Measurement on Science Learning Environment

The purpose of this research is to enrich the discussion of the component of developing instrument as a realization of student reflection on their learning environment, and as an evaluation by researchers, teachers, and principals to increase students’ motivation and achievement. The approach of the research is a quantitative research with developing the instrument method. The research is conduct on 1.067 students of public middle school in Jakarta. The result demonstrate that the instrument that developed and validated from the instrument of Classroom Emotional Climate (CEC) and the instrument of Test Of Science-Related Attitudes (TOSRA) have eight dimensions are, collaboration, motivation, care, clarity, attitude to scientific inquiry, enjoyment of lessons, adoption of scientific attitudes, and sosial implications of science, used LISREL to test Confirmatory Factor Analysis are 37 items. The further analysis use Rasch Fit Item value is 36 items and 589 person, have five items level of difficulty and person abilities are also clustered into five levels. Besides that, Cronbach's alpha value is 0.92, Person reliability is 0.88 and item reliability is 0.99, shows that the instrument results is reliabel, therefore able to use.

TOSLS Cognitive Instrument to Measure Students' Scientific Literacy Abilities

Objective: This research is for the development of the TOSLS scientific literacy assessment instrument to determine the scientific literacy abilities of undergraduate science education students. Scientific literacy skills need to be known as a form of evaluation in learning. Scientific literacy skills are one of the abilities students must have after graduating from college and facing the work environment. Method: The quantitative method used in this research uses a Rasch model approach from 29 Bachelor of Science Education program students. Indicators of item suitability with Rasch model analysis measure this quality. The questions consist of 9 multiple choice questions with a correct score of 1 and an incorrect score of 0. Data processing uses MS Excel and SPSS 25 software. Results: The results show that all fit items, according to Rasch analysis, can be maintained without revision and used as a question instrument to measure the level of science literacy ability science for students with moderate scientific literacy abilities. Novelty: The research contribution of this question instrument can be used as reference material for evaluating the lecture process in biophysics courses on fermentation topics based on the level of scientific literacy ability.

Validation of the Universal Measure of Bias–Fat (UMB Fat) among Malaysian health practitioners using Rasch analysis

ObjectiveThis study aimed to evaluate the structural validity of the Universal Measures of Bias – Fat (UMB Fat) among Malaysian healthcare practitioners using Rasch analysis. MethodsData from a cross-sectional survey of 268 public and private doctors and allied health practitioners in Peninsular Malaysia were used for this analysis. Using Rasch analysis, overall model fit and item fit of the summary UMB Fat and domain scores were examined, together with unidimensionality, response threshold ordering, internal consistency, measurement invariance, and item targeting. ResultsData showed overall misfit to the Rasch model for both the summary UMB Fat score and domain scores. Whilst unidimensionality was observed for the domain scores, this was not evident for the summary score where multiple local dependencies were present. Disordered thresholds were observed for the response format, in which the majority improved with modification. Suboptimal targeting was also detected with an uneven distribution of items at the upper and lower end of the logit scale for the summary and domain scores. Despite this, excellent internal consistency reliability was observed (person separation index: 0.76–0.89), and no measurement invariance was detected. ConclusionThe Rasch model supports reporting of the UMB Fat domain scores but not the summary score. Several issues related to local dependencies and response format were identified that could benefit from refining the UMB Fat to improve measurement accuracy, particularly when used by healthcare practitioners in Asian countries.

Psychometric properties of Mental Health Scale: An Item Response Theory Approach

Indonesia Family Life Survey (IFLS) the fifth wave used the Centre for Epidemiological Studies Scale (CES-D10) to measure mental health state, which was depressive symptoms construct. It was primarily used in some studies regarding depressive symptoms. However, there was no specific validation research in the Indonesian context. Thus, this study aimed to (1) investigate the item quality of depressive symptom measurement from IFLS fifth wave, (2) measurement precision of this scale, and (3) the measurement invariance based on gender using the IRT approach. This study used data from ILFS-5 in the KP ("Keadaan Psikologis" or Mental Health) section, which was CES-D10 consisting of 10 items and conduct IRT analysis using a 2-PL model. The number of participants from the data was 31,447. CFA analysis resulted a unidimensional model fit for this scale. Moreover, finding showed that this scale had good psychometric properties, including item discrimination, item location, and item fit, except item 5 and item 8. Despite the poor quality for those items, the reliability coefficient, including the items, met a reliable measurement criterion. Also, this scale had much information for assessing medium and severe depressive symptoms. Moreover, the Differential Item Functioning (DIF) analysis indicated that there was no item exhibiting DIF.

Validity and Reliability of Pre-Service Teacher Competency Level in Teaching Early Mathematics Instrument Using the Rasch Model

The Pre-Service Competency Teacher Level in Teaching Early Mathematics Instrument has been developed to measure the competency level and readiness to teach early mathematics in Preschool. This study was conducted to produce empirical evidence on the validity and reliability of this instrument by using the Rasch model. Surveys were conducted in three private universities that offer Early Childhood Education Programmes. A total of 85 pre-service teachers were selected as a sample of the study through simple random sampling. The validity and reliability of the instrument were measured using Rasch's model through Winstep software version 3.68.2. The findings of the unidimensionality test showed 46.1%. The Rasch analysis shows the person reliability index is 0.8 with a separation index is 2.05. The item reliability index is 0.91 and the separation index is 3.18. The item fitness analysis found 29 items in the instruments are in the valid range and one item that falls out of the allowed range. The finding showed that this instrument has a high degree of validity and reliability and can be used to measure the competency level of pre-service teaching in teaching early mathematics.

Rasch analysis, reliability, and validity of the Turkish version of the Lumbar Spine Instability Questionnaire.

Lumbar instability is an important condition that can be seen frequently in people with low back pain, affecting both the progression and the choice of appropriate exercise. The Lumbar Spine Instability Questionnaire (LSIQ) is a simple and low-cost tool for evaluating disturbed back stability in people with low back pain. The aim of this study is to develop the Turkish version of the LSIQ (LSIQ-T) and to evaluate its psychometric properties using the Rasch model. One hundred participants with chronic low back pain completed the LSIQ-T. The LSIQ-T was repeated for 30 participants after 1 week to establish its test-retest reliability. While internal and external construct validity were investigated using Rasch analysis and the Spearman correlation coefficient, respectively, reliability was evaluated in terms of internal consistency by Cronbach's alpha and the Person Separation Index (PSI). All items of the LSIQ-T were found to fit the Rasch model (chi-square: 34.07 (df = 15), p = 0.0033). The internal construct validity was good, the overall mean item fit residual was 0 (SD: 0.765), and the mean person fit residual was 0.322 (SD: 1.123). Internal consistency reliability was low with a PSI of 0.63 although Cronbach's alpha was acceptable (0.68). When the test-retest reliability was examined via differential item functioning (DIF) by time, none of the items showed DIF. The LSIQ-T is a valid unidimensional scale for the Turkish population. Although the LSIQ-T had low internal consistency, it demonstrated unidimensionality and is appropriate for use. Therefore, the LSIQ-T can be used in clinical practice and scientific research.

Translation and Validation of Mooney Problem Checklist for Students of High Schools

The study was conducted to translate and validate the Mooney Problem Checklist of high schools form. The MPCL with eleven dimensions was administered to 150 students of high schools, who were selected from government and private sector of education. Grade 9th, 10th, 11th and 12th of high schools students were considered with age range (12-18) years. This study analyzed correlation frequency and confirmative analysis. The analysis was conducted using SPSS 25th Version to examine the psychometric properties such as item fit statistics, removable items, unidimensionality, local independence, and item polarity validity. The results obtained showed that 329 items from high school form were found to have fulfilled the main assumption and measurement criteria of MPCLs. This study provides a significant contribution to improving the scale development and validation of the MPCL instrument. The results of high school students, there are 55 pairs of dimensions relationship that existed from 11 dimensions of MPCL. The results suggest that 21 pairs of dimensions have a very strong relationship, 30 have a strong and 4 pair has a moderate relationship. The findings reveal that the strongest pair MR -CTP (r= 0.892, p=0.000), has a very strong correlation. The weakest pair HPD-CSM (r=0.422, p=0.000) has a moderate correlation. This study shows the strength of the dimensions in MPCL in proving the suitability of the items used for replication in the context of schools.

The dimensionality of the Perceived Stress Scale: The presence of opposing items is a source of measurement error

IntroductionThe Perceived Stress Scale (PSS) is a widely recognized patient-reported outcome measure designed to assess an individual's level of perceived stress. The PSS consists of ten items, with six items phrased negatively and four phrased positively. Most studies have found that the PSS consists of two dimensions, with negative and positive items forming separate dimensions. However, some studies suggest a one-dimensional construct. ObjectiveWe aimed to investigate the dimensionality of the PSS and the impact of wording effects on the measurement properties of the scale. MethodsWe tested a modified version of the PSS (mPSS), with negative wording of all items. In a Danish sample, respondents were randomized to answer the PSS or the mPSS. We used confirmative factor analysis and Rasch analysis to assess the validity and reliability of the two versions. The proportion of person misfit was also evaluated. ResultsA total of 326 respondents completed the PSS, whereas 306 completed the mPSS. For the PSS, a two-factor model fitted the data best, and the first positively formulated item showed under-discrimination (Rasch model item fit rejected). The mPSS had higher measurement precision than the PSS, but evidence of local dependence was seen for both versions. Fewer respondents gave improbable responses to the mPSS compared to the PSS. LimitationsData comes from three different respondent samples. ConclusionThe PSS was deemed essentially unidimensional, with the extra variance being attributed to the reversed items. To reduce response bias, clinicians and researchers may apply the mPSS.

Finding functionality: Rasch analysis of the Functionality Appreciation Scale in community-dwelling adults in the US.

The Functionality Appreciation Scale (FAS) measures an individual's appreciation for the functions their body can perform, regardless of the individual's physical limitations. Prior studies reported on internal consistency, test-retest reliability, convergent validity, and exploratory or confirmatory factor analyses, but Rasch analysis has not yet been performed to evaluate the structural validity of the FAS. We recruited community-dwelling adults at the Minnesota State Fair and through contact lists of participants identifying interest in research done in the Brain Body Mind Lab (University of Minnesota). Community-dwelling adults with spinal cord injury (SCI) completed the FAS over Zoom. We analyzed the FAS using Rasch Measurement Theory, which produced the following outputs: item, and person fit, targeting, unidimensionality, person separation reliability (PSR), local item dependence (LID), principal component analysis of residuals (PCAR), and differential item functioning (DIF). We recruited 567 participants (average age 52.15 ± 17.5 years, 63.84% women), among which 14 adults with SCI. After rescoring 3 items and deleting 1 item, the FAS had good person and item fit (except item 4). The PCAR and subsequent paired t-tests (3.53%) confirmed the unidimensionality of the scale. There was no DIF and only one item pair had LID (item 5-6). PSR was 0.75, reflecting a capacity to differentiate groups of people with high or low functionality appreciation levels. However, there was a significant ceiling effect (28.04%) and the person mean location was 3.06 ± 2.07 logits, indicating the FAS is too easy for community-dwelling adults in the US. The 6-item Rasch-based FAS demonstrated unidimensionality, good item fit (except item 4) and person fit, but the FAS will require more difficult items to be added to improve the targeting of the scale, and better reliability.

Rasch Rating Scale Modelling of the Arabic Version of the Critical Thinking Disposition Scale

Abstract: The tendency to think critically is the motivation of an individual for using critical thinking when faced with a problem that requires a solution, making a decision or evaluating an idea. This study used the Rasch Rating Scale Model (RSM) analysis to examine a set of psychometric properties of an Arabic version of the Critical Thinking Disposition Scale (EMI): items fit, unidimensionality, local independence, equal-item-discriminations, gender differential item functioning, reliability and separation indicators and scale calibra-tion. The findings indicated that EMI showed good compatibility with the RSM as all the items matched the model except for item 11. In addition, the assumptions of the Rasch model which were unidimensionality, local independence, and equal-item-discriminations were realized. The scale had excellent reliability for persons and good reliability for items. The scale showed good separation indicators for items, and excellent separation indicators for persons. The items did not show differential gender performance. The distances be-tween the response categories were appropriate, and the category measurements showed a consistent in-crease.

Development of the Perceived Physical Literacy Questionnaire (PPLQ) for the adult population

Background/objectiveIn physical literacy (PL) research, instruments for the adult population covering all relevant domians are currently lacking in German language. Therefore, the Perceived Physical Literacy Questionnaire (PPLQ) was developed as an assessment instrument of PL for the adult population. The purpose of this study is to describe the multistage development process leading to the aim to evaluate the psychometric properties of the PPLQ. MethodsBased on established questionnaires (subscales) operationalizing the six defined PL domains (motivation, confidence, physical competence, knowledge, understanding, and physical activity behavior), we generated a large item pool. Exploratory analyses on survey data (n = 506), compelemented through an expert panel, served to identify the best fitting items. Cognitive interviews (n = 7) and a language certification process (level A2) helped to enhance the content validity of the items. Finally, we assessed the hypothesized factor structure of the PPLQ and its convergent validity with the Physical Activity-related Health Competence (PAHCO) questionnaire in a second independent sample. ResultsValid data of 417 adults (66% women, 48 ± 16 years) entered the confirmatory factor analysis. We found empirical support for a theory-compatible 24-item version, after reducing complexity (i.e., domain subscales). Additionally, the six domains could be subsumed under an overall factor for PL (χ2247 = 450.70, χ2/df = 1.83, CFIRobust = 0.895, RMSEARobust = 0.074 [CI90 = 0.063–0.085], SRMR = 0.064). Factor loadings, composite reliability, and discriminant validity were sufficient, while acceptable convergent validity was achieved for the total PL score and three domains. ConclusionThe 24-item version of the PPLQ is appropriate for assessing PL among adults. However, some items (especially in the knowledge domain) can benefit from refinement in further studies.

Panelist Analysis of Higher Education Governance Instruments Based on Good University Governance Principles Using Many Facet Rasch Measurements

This research aims to develop instruments related to measuring higher education governance so that it can fulfill the principles of good governance. This research used 3 panelists to measure the content validity of the instrument developed. The criteria for selecting experts/validators are adjusted to the expertise and needs of the instrument development being carried out. The study on testing the content validity of panelists uses Rasch Model analysis with the Facets application which aims to test subjectively rated statement items by looking at several aspects including unidimensional assumptions, testing model fit (item fit), and rater/expert consistency. The results of the content validity analysis carried out show that the unidimensional criteria (the raw variance explained by measure value is more than 20%), the item fit value (the outfit value is in the range of 0.5-1.5), and the rater consistency assessment show that all raters/experts meet the fitness criteria (outfit and infit value categories are in the range 0.5-1.5). The results of the content validity analysis carried out show that the quality of the instrument developed is suitable for use to measure higher education governance.

The assisting hand assessment for children and youth with brachial plexus birth injury: a study of validity and item hierarchy of AHA-Plex.

Functional assessments that focus on activity performance and that produce valid outcome measures for people with brachial plexus birth injury are lacking. The primary aim of this study was to re-evaluate the internal scale validity of the Assisting Hand Assessment specifically for children and adolescents with brachial plexus birth injury. Two further aims were investigating whether the scale could be shortened for this group while maintaining psychometric quality, and exploring and presenting its item difficulty hierarchy. A cross-sectional psychometric study. A convenience sample of 105 children and adolescents (aged 18 months to 18 years, mean 6 years, 7 months, standard deviation (SD) 4 years, 4 months) from Sweden, Norway, and the Netherlands with brachial plexus birth injury. Participants were assessed with the Assisting Hand Assessment. Data were analysed with Rasch measurement analysis. The 20 Assisting Hand Assessment items together measured a unidimensional construct with high reliability (0.97) and the 4-level rating scale functioned well. Item reduction resulted in 15 items with good item fit, unidimensionality, reliability and acceptable targeting. Assisting Hand Assessment for people with brachial plexus birth injury, called AHA-Plex, has 15 items and good internal scale validity. A unique item hierarchy for people with brachial plexus birth injury is presented.

Ensuring the “health” of a curricular program evaluation: Alignment and analytic quality of two instruments for use in evaluating the effectiveness of an interprofessional collaboration curriculum

To cultivate competencies in interprofessional collaboration (IPC) for patient-centered, team-based care, a multi-faceted training enhancement initiative was implemented at our academic primary care residency site. Evaluation of the activities from previously collected survey data occurred upon a 2-year review. First, the evaluation team scrutinized the instruments for alignment and appropriateness with planned IPC educational learning and behavior objectives. We found the two instruments were well supported by the literature and with appropriate evidence for validation, but were not well aligned to the objectives of this IPC training initiative, reducing appropriateness of potential inferences of the findings for this context. Second, the team assessed the analytic quality of survey results in item difficulty distribution and item fit to the requirements of a Rasch measurement model. This revealed low person separation due to high overall item agreement. Most residents agreed with most items, so the measures lacked the precision necessary to capture change in residents’ IPC competency. Our instrument review serves as a reminder of the need to gather validity evidence for the use of any existing tool within a new context, and offers a generalizable strategy to evaluate data sources for appropriateness and quality within a specific program.

Rasch 모형을 이용한 뷰티전문가의 자기관리 측정도구 타당화

The purpose of this study was to validate the self-management measurement tool's validity for beauty professionals using the Rasch model. Data was collected from 679 participants who were beauty professionals working in the metropolitan area. Kim Hyun-jeong's (2021) self-management scale was used as the measurement tool. Data analysis methods included testing for normality, exploratory factor analysis, and confirmatory factor analysis. In the Rasch model analysis, the validation of the data included testing for unidimensionality, item fit and difficulty, and the validity of response category thresholds.
 The results obtained from the research are as follows. First, the normality testing of the self-management measurement tool's sub-factors for beauty professionals confirmed that the data showed no skewness and was within the normal distribution range. Second, both the exploratory factor analysis and confirmatory factor analysis of the self-management measurement tool for beauty professionals confirmed that all items were appropriate, thus validating its construct validity. Third, the analysis of item fit using the Rasch model confirmed that the self-management measurement tool for beauty professionals established validity in terms of unidimensionality, item fit and difficulty, and the appropriateness of item category thresholds.
 In summary, based on the comprehensive findings of this study, Kim Hyun-jeong's (2021) self-management measurement tool for beauty professionals was found to be highly valid.

AN ENTREPRENEURIAL CREATIVE THINKING TEST FOR HIGH SCHOOL STUDENTS IN STEM EDUCATION

Instruments have been developed to measure entrepreneurial thinking. However, those instruments did not assess the commercial aspects of product marketing using digital technology. Consequently, the Entrepreneurial Creative Thinking Test (ECTT) was developed to assess the level of entrepreneurial creative thinking among Form Four high school students in STEM education. ECTT consists of five constructs derived from integrating the steps between the Socioscientific Issue Approach Model and the Design Thinking Process Model. The five constructs are Investigation, New Idea, Design, Create, and Commercial. The sample consisted of 187 Form Four students aged 15 to 16 years from four high schools in Sabah, Malaysia. The sample comprised 103 females (55.1%) and 84 males (44.9%). A survey research design was conducted to ascertain the extent to which the validity and reliability of this instrument can meet the psychometric characteristics of a research. The survey research design focused on the validity and reliability assessment of ten open-ended questions. This study utilised the Rasch Measurement Model to evaluate the validity and reliability of the ECTT instrument. Person fit, item fit, item polarity, unidimensionality, reliability, and item-person separation were analysed. The study's findings revealed that the ECTT instrument has good item reliability, with a score of .97. In conclusion, the ECTT is a valid and reliable instrument for assessing the level of entrepreneurial creative thinking among Form Four students in STEM education.

Standard Assessment Model for Scientific Integration-Interconnection as Material for Evaluation of Vision and Mission Achievement

This study aims to develop a standardized assessment model of scientific integration-interconnection, which begins with conducting a literature review and needs assessment, setting goals, developing initial products, validating contents, testing readability, testing instruments, analyzing results, and ending with an interpretation of the results of the assessment. The method used is Research and Development. Testing of the assessment instruments was carried out for each element of the Tridharma of Higher Education, namely 76 lesson plans, 139 research and development works, and 41 community service works. The scientific integration-interconnection assessment model has characteristics according to the structure of the second order model, with the first order being 4 scientific integration-interconnection domains (philosophical domain, material domain, methodological domain, and strategic domain) and the second order is 15 aspects of scientific integration-interconnection. Data analysis techniques using Aiken-V, Exploratory Factor Analysis/EFA and Confirmatory Factor Analysis/CFA, Omega, Item Response Theory approach of polytomous scoring, and descriptive statistics. The assessment model is feasible, fulfilling the criteria of content validity, construct validity (p-value = 0.11949 and RMSEA = 0.027), estimated instrument reliability (α = 0.816), and construct reliability (ɷ =0.939). The scoring used the GRM approach with good quality in terms of the results of the 94% item fit test with an item discrimination index of 0.655 < DISCi < 1.153 and the average item difficulty level of -1.965 < DIFFi < -0.662.