Rating Scale Categories Research Articles

Exploring the Effects of Collapsing Rating Scale Categories in Polytomous Item Response Theory Analyses: An Illustration and Simulation Study

ABSTRACT Researchers who work with ordinal rating scales sometimes encounter situations where the scale categories do not function in the intended or expected way. For example, participants’ use of scale categories may result in an empirical difficulty ordering for the categories that does not match what was intended. Likewise, the level of distinction between categories may be imprecise, where there are only minimal differences between categories or categories that reflect too wide of a range of the latent variable to be useful. In these cases, researchers sometimes consider combining categories to minimize the impact of these characteristics on the interpretability of scores. Although this practice is relatively common, the psychometric literature provides limited guidance related to specific strategies for and the psychometric consequences of combining categories. We used empirical data from a self-efficacy scale to illustrate rating scale malfunctioning in a real-world context and demonstrate the impact of various category-collapsing schemes on indicators of rating scale functioning. Then, we used a simulation study to consider how these category-collapsing schemes may impact rating scale functioning under different conditions. Overall, our analyses illustrate different approaches to category collapsing and their impact on psychometric results. Our results suggested that aligning the collapsing scheme with the location of the rating scale malfunctioning was generally effective in improving scale performance. We discuss implications for research and practice and identify several directions for further investigation.

Item-Level Psychometrics of the Neonatal Eating Outcome Assessment in Orally Feeding Infants.

The Neonatal Eating Outcome Assessment determines feeding performance based on the infant's postmenstrual age (PMA). To examine item-level measurement properties of this assessment's rating scale. In this retrospective study, Rasch analysis was completed on clinical data from the Neonatal Eating Outcome Assessment for 100 infants (52 preterm and 48 full-term) using Winsteps version 3.93.1. Instead of PMA-based scores, ordered letters converted to numerical scores were analyzed. Analysis demonstrated that Section I (Pre-Feeding Skills) represents a separate construct from Sections II and III (Oral Feeding and End of Feeding, respectively). Sections II and III were adequately unidimensional to complete Rasch analysis. These sections fit the Rasch model overall, but rating scale category underuse was common, which may be attributed to sample characteristics. This analysis supports using validated ordered letter scoring of Sections II and III to measure oral feeding performance in preterm and full-term newborns.

Applying Rasch analysis in refinement and validation of interpersonal skills measure for gifted children.

Interpersonal characteristics of gifted adolescents is important because of the potential influence on individuals' psychological health and future professional success. Understanding the interpersonal characteristics requires valid and reliable measures. This study attempted to explicitly describe the application of Rasch analysis in the validation and development process of an existing measure of interpersonal skills among gifted adolescent in Ohio. We extensively evaluated the psychometric properties of the 40-item scale measuring the interpersonal competence of adolescents among the gifted population in Ohio (N = 127) using Rasch analysis. Multiple aspects of reliability and validity were tested including dimensionality, rating scale functioning, and fit statistics. The internal consistency reliability of the scale was confirmed with an adequate fit to the Rasch model. However, the scale demonstrates relatively poor performance in terms of unidimensionality with our sample. Also, the rating scale categories were confusing given that our sample could not effectively distinguish some adjacent categories. Corresponding refinements have been made and the revised scale formed a meaningful linear progression with improved performance on unidimensionality, rating scale functioning, and fit statistics. The study provided evidence that the construct of interpersonal skills is measurable. Based upon the original Interpersonal Competence Questionnaires consists of 40 items, the author extracted and piloted a refined measure consisting of 31 items that performed a meaningful, theoretically consistent linear progression measure that could be used to measure the level of interpersonal skills of gifted children.

Kondisi Kejenuhan Belajar Pada Siswa Kelas X SMA Islam Jepara

<p class="ContentAbstrak">This research is a study of the saturation condition of learning at Jepara Islamic High School. This study aims to determine the level of learning saturation conditions that exist in Jepara Islamic High School. This research was conducted by distributing questionnaires to be filled out by students there. The method used to analyze the data is a quantitative descriptive analysis technique which is included in the score distribution and rating scale categories that have been determined. The results of the validation of learning saturation are: (1) student learning saturation is high with a percentage of 59%. (2) student learning saturation is low with a percentage of 41%. Based on these results, the level of learning saturation of class X students at Jepara Islamic High School is in the fairly high category.</p>

Pengembangan LKPD Berbasis Web pada Materi Kingdom Animalia untuk Meningkatkan Hasil Belajar Siswa

This study aims to develop Web-based LKPD on animalia material, to find out the improvement of students' biology learning outcomes through the use of valid, practical, and effective Web-based LKPD. This study uses a development research approach (Research and Development). The object of this research is a web-based LKPD on the material kingdom animalia. The validation and trial stages were carried out on the feasibility of Web-based LKPD. Validation by material experts was carried out to obtain advice regarding the feasibility of web-based worksheets in terms of material. The limited trial was carried out using the response questionnaire method from students in class X MIPA 3 SMA Negeri 1 Wajo. The data analysis technique uses a quantitative descriptive analysis technique with a score of five (Likert scale) against a predetermined rating scale category. The results of the study were obtained, the results of the effectiveness test of limited and wide-scale learning outcomes from the results of the T test with a value of α <0.01 so that they were declared to have a significant effect. The N-Gain test result is 0.75 in the "Low" category. In conclusion, the results of the study show that the development of web-based worksheets for biology subjects in Kingdom Animalia can improve the learning outcomes of SMA Negeri 1 Wajo students, especially in XMIPA 3 class. 
 
 Keywords: Kingdom Animalia, LKPD Development, Web.

Validating criteria for identifying core concepts using many-facet rasch measurement

IntroductionCore concepts are foundational, discipline-based ideas considered necessary for students to learn, remember, understand, and apply. To evaluate the extent to which a concept is “core,” experts often rate concepts using various criteria, such as importance, complexity, and timeliness. However, there is a lack of validity evidence for core concepts criteria.MethodsUsing a three-facet Many-Facet Rasch Measurement (MFRM) model, we analyzed 7,558 ratings provided by 21 experts/raters for 74 pharmacology core concepts using five criteria measured on a five-point scale.ResultsAll of the criteria had Infit or Outfit MnSq values within the expected range (0.5 < MnSq < 1.7), suggesting the criteria contained an acceptable amount of variability; a reliability index of approximately 1.00 suggested that the criteria were reliably separated with a high degree of confidence. The rating scale Outfit MnSq statistics also fell within the 0.5–1.7 model fit limits; the “average measure” and “Rasch-Andrich thresholds” increased in magnitude as the rating scale categories increased, suggesting that core concepts with higher ratings were in fact meeting the criteria more convincingly than those with lower ratings. Adjusting expert ratings using the MFRM facets (e.g., rater severity) resulted in reorganization of core concepts rankings.ConclusionThis paper is a novel contribution to core concepts research and is intended to inform other disciplines seeking to develop, implement, and refine core concepts within the biomedical sciences and beyond.

Psychometric Characteristics of the Kalam Cognitive Test (KCT): An Andrich Rating Scale Analysis

The science of Kalam is one of the compulsory courses for prospective Islamic teachers, and mastery of this material is very important. Therefore, the measurement instrument used must have good psychometric characteristics to accurately reflect the abilities of prospective Islamic teachers. The purpose of this study is to analyze the psychometric characteristics of the Kalam Cognitive Test (KCT), which has been developed with the Andrich Rating Scale and assisted by the Winsteps program. The research, which was conducted in the spring semester of the 2022–2023 academic year, involved 44 prospective Islamic religion teacher students in the Islamic Religious Education Study Program at Jambi Ma'arif Islamic College who were selected through cluster random sampling techniques. The results of the analysis show that the KCT has good psychometric characteristics in terms of fulfilling the assumptions of the analysis, item reliability, person reliability, and item difficulty level. Related to the rating scale analysis, it can also be concluded that all rating scale categories have met the required criteria so that they have functioned properly.

Assessing individual-level measurement precision of the Short Physical Performance Battery using the test information function

The Short Physical Performance Battery (SPPB) is a commonly used tool to assess lower extremity function, composed of three assessments (standing balance, gait speed, and chair stand). While its validity for group-level decisions has been positively demonstrated, the measurement precision at the individual level needs more clarification. We aimed to examine the SPPB's psychometric characteristics including its conditional measurement precision with Rasch methods in a sample of elderly patients admitted to cardiac rehabilitation. We analyzed prospectively collected SPPB data from 637 patients aged ≥75 years, admitted to our cardiac rehabilitation department (January 2018-July 2019). After classical Rasch analysis, we calculated the test information function to examine the measurement precision of the SPPB along the score continuum. The mean SPPB score was 6.3 points (SD 3.4) (potential range 0-12). Our results confirmed the SPPB unidimensionality (variance attributable to the main factor: 84.8%), appropriate functioning of rating scale categories, internal construct validity (infit and outfit mean-square statistics: 0.90-1.09), and no item local dependence (residual correlations <0.2). However, the measurement precision at the individual level was quite limited (SE > 0.94 logits): the confidence intervals for true scores were, at best, about one-third of the score range (four points). This limited measurement precision increases the risk of inappropriate clinical decisions about individuals in diagnostic classification or change assessment. Therefore, further research of high methodological quality is warranted on this point. Moreover, a clearer distinction between group-level vs. individual-level indicators of change is necessary.

DEVELOPING DIGITAL BOOK BASED FLIP PDF PROFESSIONAL INTEGRATED WITH LIVE WORKSHEET TO INCREASE STUDENT MOTIVATION FOR LEARNING WRITING IN THE FIRST GRADE SMPN 5 KARANGPLOSO IN 2022

This study aims to develop a Digital Book that is integrated with an interactive worksheet (Liveworksheet) in English subjects and to determine the effect of using digital books on student motivation at SMP Negeri 5 Karangploso. This study uses a research and development approach (Research and Development). The place of this research is SMP Negeri 5 Karangploso. The object of this research is a Fun Digital Writing Book "Yes, I Can Write" which is a digital book that is integrated with an interactive worksheet (Liveworksheet). This research is included in the type of Research and Development (R&amp;D). The data collection technique used in this research is the interview and questionnaire method. Interviews were used to analyze the needs of students in learning English, then the second questionnaire was used to collect data about the feasibility of English learning media, including: material experts, media experts and class VII E students. The method used to analyze the data was expressed in a four-scale score distribution. to a predetermined rating scale category. From the research results, among others: the material expert test obtained a score of 80.5% in the very good category, the media expert test obtained a score of 93.1% in the very good category, and the test to students with a score of 88% in the very good category so Fun Digital Writing Book “Yes, I Can Write” can be used to increase students' learning motivation in writing skills in English.

Determining the Hierarchy of Coma Recovery Scale-Revised Rating Scale Categories and Alignment with Aspen Consensus Criteria for Patients with Brain Injury: A Rasch Analysis.

This study aimed to empirically evaluate the hierarchical structure of the Coma Recovery Scale-Revised (CRS-R) rating scale categories and their alignment with the Aspen consensus criteria for determining disorders of consciousness (DoC) following a severe brain injury. CRS-R data from 262 patients with DoC following a severe brain injury were analyzed applying the partial credit Rasch Measurement Model. Rasch Analysis produced logit calibrations for each rating scale category. Twenty-eight of the 29 CRS-R rating scale categories were operationalized to the Aspen consensus criteria. We expected the hierarchical order of the calibrations to reflect Aspen consensus criteria. We also examined the association between the CRS-R Rasch person measures (indicative of performance ability) and states of consciousness as determined by the Aspen consensus criteria. Overall, the order of the 29 rating scale category calibrations reflected current literature regarding the continuum of neurobehavioral function: category 6 "Functional Object Use" of the Motor item was hardest for patients to achieve; category 0 "None" of the Oromotor/Verbal item was easiest to achieve. Of the 29 rating scale categories, six were not ordered as expected. Four rating scale categories reflecting the Vegetative State (VS)/Unresponsive Wakefulness Syndrome (UWS) had higher calibrations (reflecting greater neurobehavioral function) than the easiest Minimally Conscious State (MCS) item (category 2 "Fixation" of the Visual item). Two rating scale categories, one reflecting MCS and one not operationalized to the Aspen consensus criteria, had higher calibrations than the easiest eMCS item (category 2 "Functional: Accurate" of the Communication item). CRS-R person measures (indicating amount of neurobehavioral function) and states of consciousness, based on Aspen consensus criteria, showed a strong correlation (rs = 0.86; p < 0.01). Our study provides empirical evidence for revising the diagnostic criteria for MCS to also include category 2 "Localization to Sound" of the Auditory item and for Emerged from Minimally Conscious State (eMCS) to include category 4 "Consistent Movement to Command" of the Auditory item.

Evaluating the Psychometric Properties of a Proposed Farsi Version of the Interpersonal Reactivity Index Using Item Response Theory

Purpose Although researchers have examined empathy among many populations worldwide, investigations of empathy among Farsi-speakers are limited. The purpose of this study is to evaluate the psychometric properties of the Interpersonal Reactivity Index (IRI) for Farsi-speakers (IRI-Farsi). Methods After translating, we explored psychometric properties of the IRI-Farsi with exploratory factor analysis and item response theory using a sample of Iranians ( N = 517). Results The IRI-Farsi appeared to exhibit a four-factor structure and acceptable item properties within each subscale. Moreover, the IRI-Farsi rating scale categories were generally ordered and distinct with emotion-triggering items as easier to endorse compared to more complex cognitively aroused statements. Conclusions Results support using the IRI to measure dispositional empathy in mainland Iran. Social work researchers can use these results to inform research and practice related to empathy in this population and design more effective interventions to increase awareness of empathic feelings and understanding for practitioners and clients.

Evidence for validity of the Swedish self-rated 36-item version of the World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) in patients with mental disorders: a multi-centre cross-sectional study using Rasch analysis

BackgroundThe World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is a generic instrument for the assessment of functioning in six domains, resulting in a total health-related disability score. The aim of this study was to investigate the psychometric properties of the Swedish-language version of the self-rated 36-item version in psychiatric outpatients with various common psychiatric diagnoses using Rasch analysis. A secondary aim was to explore the correlation between two methods of calculating overall scores to guide clinical practice: the WHODAS simple (summative) model and the WHODAS complex (weighted) model.MethodsCross-sectional data from 780 Swedish patients with various mental disorders were evaluated by Rasch analysis according to the partial credit model. Bivariate Pearson correlations between the two methods of calculating overall scores were explored.ResultsOf the 36 items, 97% (35 items) were within the recommended range of infit mean square; only item D4.5 (Sexual activities) indicated misfit (infit mean square 1.54 logits). Rating scale analysis showed a short distance between severity levels and disordered thresholds. The two methods of calculating overall scores were highly correlated (0.89–0.99).ConclusionsThe self-administered WHODAS 2.0 fulfilled several aspects of validity according to Rasch analysis and has the potential to be a useful tool for the assessment of functioning in psychiatric outpatients. The internal structure of the instrument was satisfactorily valid and reliable at the level of the total score but demonstrated problems at the domain level. We suggest rephrasing the item Sexual activities and revising the rating scale categories. The WHODAS simple model is easier to use in clinical practice and our results indicate that it can differentiate function among patients with moderate psychiatric disability, whereas Rasch scaled scores are psychometrically more precise even at low disability levels. Further investigations of different scoring models are warranted.

Development and validation of the symptom burden questionnaire for long covid (SBQ-LC): Rasch analysis

ObjectiveTo describe the development and validation of a novel patient reported outcome measure for symptom burden from long covid, the symptom burden questionnaire for long covid (SBQ-LC).DesignMultiphase, prospective mixed methods...

Validation of the 0-10 verbal numeric scale for assessment of pruritus severity in dogs.

The pruritus Visual Analog Scale (pVAS) is currently the only validated tool for assessing canine pruritus. A verbal numeric scale (VNS) offers an alternative if clients are not present or able to complete the pVAS. To validate the 0-10 VNS and evaluate its interchangeability with the pVAS. 251 dogs were included in the study. Dog owners were asked to assess their pet's pruritus level using the pVAS, the 0-10 VNS and the verbal rating categories (mild, moderate, severe) in one or two subsequent visits. To be valid, VNS must satisfy the following: correlate with pVAS (criterion validity); indicate reduction in itch score after an antipruritic intervention (construct validity); and have scores related to the verbal rating categories that are significantly different (content validity). VNS and pVAS interchangeability and the percentage of clients that preferred the VNS and pVAS also were evaluated. The VNS and pVAS correlation for visits 1 and 2 was good (rICC = 0.9) and excellent (rICC = 0.94). The VNS scores were significantly decreased after antipruritic interventions (P < 0.001). The VNS scores associated with the verbal rating scale categories (mild, moderate and severe) were significantly different (P < 0.001). The 95% limits of agreement for the pVAS and VAS were outside the limit of acceptability of ±2 (-2.0, 2.57). Most clients (69%) preferred the pVAS to the VNS. The VNS is a valid scale to evaluate canine pruritus; however, the VNS and pVAS are not interchangeable.

Applying the Rasch Rating Scale Model (RSM) to investigate the rating scales function in survey research instrument

The functionality and quality structure of a rating scale are vital in a survey instrument. This article discusses the underlying assumption of a rating scale and addresses the application of the Rasch Rating Scale Model (RSM) to diagnose the rating scale structure in a survey instrument. The instrument used to demonstrate the process of diagnosis a rating scale functioning in this study was the Vocational Teachers’ Assessment Literacy (VoTAL) instrument. The VoTAL instrument utilized the five-point Likert-type response format and consisted of 88 items representing three assessment literacy constructs. The data were obtained from 224 vocational teachers at five vocational colleges, which were randomly selected from the state of Selangor and the federal territory of Kuala Lumpur in Malaysia. The rating scale diagnosis results showed that the initial five steps of rating scale categories were not properly functioning as intended. It was found that the Andrich threshold of category three was disordered as it was not monotonically advance with categories. Additionally, the result also found that the width of the threshold between category three and category four was too narrow (0.66 logits). The results indicate that the rating scale used in the VoTAL instrument was disordered. Thus, it was suggested that the initial five rating step categories were collapsed down into four categories. In conclusion, this study highlights the importance of assessing the rating scale’s functionality in a survey instrument to reduce measurement error and collect valid and reliable data.

Alliffa HUBUNGAN UPAH DAN TINGKAT KESEJAHTERAAN PEMAIN BOLA VOLI DALAM MENGIKUTI PERTANDINGAN ANTAR KAMPUNG (TARKAM) DI KOTA SEMARANG

The purpose of this study was to determine the needs and welfare level of volleyball players in participating in inter-village matches. This study uses a mixed method research, namely in conducting research the researcher uses a combination of quantitative methods and qualitative methods. Methods of data collection using questionnaires, observation, documentation, and interviews. The research instrument used a questionnaire (questionnaire). The data analysis technique uses quantitative descriptive analysis techniques which are expressed in the distribution of scores and percentages to predetermined rating scale categories. The results showed that according to the sub-indicators of physiological needs, security needs, ownership needs, reward needs, self-activation needs, the welfare level of volleyball players in participating in inter-village competitions (tarkam) in Semarang City, the majority are in the high category. This shows that volleyball players who participate in inter-village competitions have been fulfilled from all sub indicators ranging from athlete's physiology, safety when competing, having good needs, awards awarded according to the efforts of athletes, and athlete's self-actualization

Pengembangan E-Book Interaktif sebagai Media Tambahan Pembelajaran Bahasa Inggris di Kelas X SMA

This study aims to develop interactive electronic book learning media (E-book) in English subject and determine the feasibility of electronic book learning media (E-book). This study uses a Research and Development approach. The setting of this research is in SMA Darul Karamah Singosari. The object of this research is an interactive electronic book of English subject. The data collection technique is a questionnaire method, which is used to analyze the needs of students in learning English online, then the second questionnaire is used to collect data about the feasibility of English learning media, including: material expert and media expert. The method of data analysis is expressed in the distribution of scores of five scales to a predetermined rating scale category. From the research results, among others: the material expert test obtained a score of 76% in good category, the media expert test obtained a score of 78% in good category, so that the interactive e-book learning media in English subjects is appropriate in the teaching and learning activities for the students of Tenth Grade Senior High School

The Development of Islamic Big Book Learning Media For Early Children's Languages

This article aims to develop Big Bookis learning media (Big Book Islami). The problems that occur are the children's language skills are still not optimal and the learning media used by the teacher is less attractive so that they are less able to improve children's language skills. This article uses a research and development approach. The data collection technique used in this research is a questionnaire and documentation method. Media that have been assessed by material experts and teachers are then given to students of Kindergarten ABA 13 Helvetia Medan as respondents and their language development is assessed by the teacher. The method used to analyze the data is expressed in the distribution of scores on a scale of five against a predetermined rating scale category. The results of Islamic Big Book Learning Media Development are declared suitable for use after being tested by developers, material experts, media experts, and users. Details of the data obtained from the test results include: material expert testing obtained a score of 4 and a presentation value of 80% with a decent category, media expert testing obtained a score of 4, and a presentation value of 80% with a decent category, and the results of children's language development through Islamic Big Book 80 media. .83 with a well-developed category. The results of the study show that Big Bookis media can improve children's language skills in Kindergarten ABA 13 Helvetia Medan.

Evaluating the psychometric properties of the Grit scale in Marine recruits using Rasch analysis

IntroductionSuccessful completion of initial military training has been suggested to be predicted by physical abilities, cognitive abilities and non-cognitive abilities such as hardiness and grit. This study aimed to assess...

Psychometric evaluation of the Arabic version of the Quebec user evaluation of satisfaction with assistive technology (A-QUEST 2.0) in prosthesis users.

The evaluation of patient satisfaction and perceptions plays a vital role in determining the quality of prosthesis users' devices and the competency of healthcare services. To evaluate the psychometric properties of the Arabic Quebec User Evaluation of Satisfaction with Assistive Technology (A-QUEST 2.0) with prosthetics users. A methodological study. Saudi Arabia, Turkey. A convenience sample of outpatient prosthesis users (N.=183). The A-QUEST 2.0 includes two subscales respectively evaluating the user's satisfaction with the device and the services provided. The data for each subscale were investigated using Rasch analysis to evaluate the item fit, reliability indices, item difficulty, local item dependency, and differential item functioning (DIF). Both subscales met the Rasch criteria for the functioning of rating scale categories. All items showed an acceptable fit to the Rasch model. The person separation indices for the Device and Services subscales were 2.21 (Cronbach's α=0.90) and 1.72 (Cronbach's α=0.85), respectively. Therefore, the two subscales are sensitive enough to distinguish between at least three different levels of satisfaction. The unidimensionality of each subscale was confirmed, and none of the items displayed differential item functioning across age, gender, location of amputation, country, and duration of use. Overall, the findings indicate the psychometric evaluation of A-QUEST 2.0 is effective with prosthesis users across different clinical contexts and cultures. Thus, the A-QUEST 2.0 allows for a comprehensive understanding of users' perceptions of prosthesis characteristics, particularly among subjects with lower limb amputations caused by traumatic injuries. Our paper provides clinicians dealing with Arabic patients a validated outcome measure for satisfaction with prosthesis. Besides providing information in the development of new products and service delivery. Further studies are necessary to improve the measure's metric quality in different contexts and for different prosthesis devices.