Rasch Measurement Approach Research Articles

Validation of the Online Test Anxiety Inventory (ON‐TAI) for adult students. The Rasch measurement approach

AbstractThis study is an investigation of the validity of the Online Test Anxiety Inventory (OΝ‐TAI) for adult students. The scale contained the 20 items of the Test Anxiety Inventory (Spielberger, 1980), together with five computer anxiety items, all rephrased so as to pertain to online test anxiety. The scale was administered to a large sample of Greek and Cypriot postgraduate students and Rasch measurement was used for the investigation of its psychometric properties. After original analyses, one of the computer anxiety items was removed, leaving the final instrument with a total of 24 items. Results showed that the scale is unidimensional thus supporting that computer anxiety can indeed be considered a component of online test anxiety. All evidence collected supports the high degree of construct validity of the scale. Furthermore, results verify that the 4‐point scale used is optimal. Additional research, with adult student populations from other countries, with the use of the OΝ‐TAI is suggested for the purpose of broader use.

The metacognitive awareness of reading strategy among pre-service primary teachers and the possibility of rating improvement using Rasch analysis

This study investigated pre-service primary teachers' reading strategies using the Metacognitive Awareness of Reading Strategy Inventory (MARSI) in the Indonesian version. The participants were 4377 pre-service primary teachers in Indonesia. The data were collected via an online form, and the invitations were sent out randomly. The participants were divided into two studies: 2912 participants were in study 1, which used a 5-point Likert scale, and 1465 participants were in study 2, using a 4-point Likert scale. The Rasch measurement approach was used in investigating the validity and reliability, rating improvement analysis, checking item difficulty functioning based on gender, and classifying pre-service primary teacher reading strategies. The adapted questionnaire satisfied all psychometric requirements based on Rasch modeling. The questionnaire with a 4-point scale showed better function than the 5-point one based on the observed measure average and curve probability category. The teachers' abilities ranged from − 6 logit to 6.45 logit.

Detecting Differential Rater Severity in a High-Stakes EFL Classroom Writing Assessment: A Many-Facets Rasch Measurement Approach

Detecting Differential Rater Severity in a High-Stakes EFL Classroom Writing Assessment: A Many-Facets Rasch Measurement Approach

Retracted: Arab and Malay Students’ Attitudes toward Statistics and Their Learning Styles: A Rasch Measurement Approach

Retracted: Arab and Malay Students’ Attitudes toward Statistics and Their Learning Styles: A Rasch Measurement Approach

Building an Initial Validity Argument for Binary and Analytic Rating Scales for an EFL Classroom Writing Assessment: Evidence from Many-Facets Rasch Measurement

Although much research has compared the functioning between analytic and holistic rating scales, little research has compared the functioning of binary rating scales with other types of rating scales. This quantitative study set out to preliminarily and comparatively validate binary and analytic rating scales intended for use in formative assessment and for paragraph writing assessment in a Thai EFL university classroom context. Specifically, this study applied an argument-based validation approach to build an initial validity argument for the rating scales with emphasis on the evaluation, generalization, and explanation inferences, and employed a many-facets Rasch measurement (MFRM) approach to investigate the psychometric functionalities of the rating scales which served as the initial validity evidence for the rating scales. Three trained teacher raters applied the rating scales to rate the same set of 51 opinion paragraphs written by English-major students. The rating scores were analysed following the MFRM psychometrics. Overall, the MFRM results revealed that (1) the rating scales largely generated accurate writing scores, supporting the valuation inference, (2) the raters were self-consistent in applying the rating scales, contributing to the generalization inference, (3) the rating scales sufficiently captured the defined writing construct, substantiating the explanation inference, and (4) the binary rating scale showed more desirable psychometric properties than the analytic rating scale. The present findings confirm the appropriate functioning and reasonable validity argument of the rating scales and highlight the greater potential of the binary rating scale to mitigate rater inconsistency and cognitive load in a formative classroom assessment.

Arab and Malay Students’ Attitudes toward Statistics and Their Learning Styles: A Rasch Measurement Approach

Students’ learning of statistics has been studied from a variety of angles, and this study is no different. The main purpose is to compare the Malay and Arab students’ attitudes toward learning statistics and their learning styles in understanding statistics. A survey questionnaire and face-to-face interview techniques were used to elicit information from 150 students based on the cohort. They were asked about how they learn to solve statistical problems based on Kolb’s four learning cycles: feeling, reflective observation, thinking, and doing. Attitude responses were numerically recorded based on a five-point Likert scale, while preference for learning styles was recorded as 1 (Do and Feel) or 0 (Watch and Think). Both attitude and learning style data were combined and subjected to Rasch analysis. Results show that a majority of the Arab and Malay students have moderate to high positive attitude toward learning statistics. Generally, students from both cultures are classified as the “Accommodating” type with a preference for doing and feeling from the experience of doing statistical problems. Arab students are classified as the “Assimilating” type with a preference for thinking, reflecting, and learning from observation, while Malay students are classified as the “Converging” type with a preference for thinking and doing statistical exercises.

Development of ADiagnostic Assessment Test to Evaluate Science Misconceptions in Terms of School Grades: A Rasch Measurement Approach

Development of ADiagnostic Assessment Test to Evaluate Science Misconceptions in Terms of School Grades: A Rasch Measurement Approach

Analysis of Validity and Reliability of Economic Achievement Test Based on Rasch Measurement Model

Teachers regularly attend assessments on students to identify students’ levels of mastery. Achievement tests as a quality measurement tool are quintessential so that the conclusions obtained are reliable and significant. Therefore, high-quality achievement tests need to satisfy specific criteria by going through standard procedures. Nonetheless, time and competency constraints lead teachers to utilise economic test questions that do not reach specific standards. Therefore, this research intended to develop and test the validity and reliability of economic achievement tests. An economic achievement test instrument was developed, consisting of 30 objective questions based on Bloom’s taxonomy. The testing of the instrument involved 40 respondents of Form Six economics students. The researchers appointed five experts to evaluate the validity of the content of the achievement test questions. At the same time, the construct validity and instrument reliability test analysis involved item-respondent reliability analysis, item-respondent separation index, Cronbach’s alpha, item polarity, item fit, standardised residual item correlation and respondent item-ability difficulty level distribution using Rasch measurement approach through Winsteps software 3.72.3. The data of the tests conducted determined that the achievement test confirmed good content validity and reliability values. The analysis also established that six questions needed to be modified. The development of the economic achievement test offers an alternative measurement design over future performance test testing. The researchers proposed that the implementation of this measurement on other subjects too.

Measuring Social and Emotional Wellbeing in Aboriginal Youth Using Strong Souls: A Rasch Measurement Approach.

Currently, there are few robustly evaluated social and emotional wellbeing (SEWB) measures available for use with Aboriginal youth in research, policy, and practice. As such, this study used a Rasch measurement approach to examine the psychometric properties of Strong Souls, a 25-item self-reported SEWB instrument, created for use with Aboriginal youth in the Northern Territory. Our sample (N = 154) included youth (15–25 years old) living on Whadjuk (metropolitan Western Australia; N = 91) and Kamilaroi countries (rural New South Wales; N = 63). Using Rasch modelling techniques, evidence for multidimensionality in the scale was observed, resulting in subsequent analyses conducted separately on two subscales: Psychological Distress and Resilience. The Resilience subscale did not meet the Rasch model assumptions, with poor person and item separation and reliability indexes suggesting the scale was not reliably differentiating between participants’ Resilience scores. The Psychological Distress subscale had mixed separation and reliability index results, with good construct validity implied but poorer ability to target the distress of participants. Our findings provide novel evidence demonstrating the functioning of Strong Souls in a contemporary sample of Aboriginal youth, suggesting further modifications of the instrument are required before it can be used with confidence as a reliable measure in this population group.

Validation of modified youth experience survey for sport (MYES-S) using multidimensional Rasch model

ABSTRACT Youth sports provides the foundation for successful dynamics and organization within a sports ecosystem. Thus, it is necessary to explore the experiences of youth who either drop out of or persist in a sport. The study’s main goal is to provide empirical evidence of the validity and reliability of the modified Youth Experiences Survey for Sport (MYES-S), which assesses the personal experiences of youths in sports activities using the multidimensional Rasch model. The present study is a first-of-its-kind attempt to introduce a multidimensional Rasch measurement approach into the sports psychology field while still addressing the limitations of the classical test theory approach. A total of 1,199 athletes from ninth through 12th grade participated in the survey. The findings support a six-dimensional model of the MYES-S. The person separation reliability estimates for each dimension were close to the acceptable range. Most student-athletes’ logit scores were above the range of the item endorsibilities except in cognitive skill (CS); thus, more items need to be added to better capture the youth experience in sports. Future research directions include addressing differential functioning items across gender and race groups and having more appropriate response categories. The significance of this study lies in its potential to contribute to the investigation and the calibration of the MYES-S to provide practical and meaningful guidance to sports psychology researchers and practitioners vis-à-vis the understanding of the youth sports experience.

The effect of gender on teaching dispositions: A Rasch measurement approach

The gender imbalance in the teaching profession occurs worldwide and declining male representation is a cause of concern. While some studies have looked into causes to explain disproportionate male representation, the notion that females may possess higher levels of teaching dispositional traits than males has not been empirically pursued in any depth. To address this, we investigated the effect of gender on 324 first year pre-service teachers’ dispositions toward teaching. Using the Teacher Disposition Scale (TDS) (West et al., 2018) to measure teaching dispositions, a Rasch analysis (Andrich, 1988) on the data indicated females were significantly more disposed toward being effective teachers than males on the core teacher dispositional traits of teacher efficacy, and interpersonal and communication skills. These findings provide initial evidence using a rigorous psychometric approach in the understanding of the significant effect of gender on teaching dispositions.

Measuring Students’ Understanding in Probability Concept between Two Different Cohort using Rasch Measurement

The students’ level of proficiency in any particular course is individually distinctive. Therefore, it is necessary for the educators to be able to address their student’s level of ability in understanding of the course they enrolled. Particularly, educators should be able to design a set of questions which suits the level of their students’ ability. For this reason, this study is concentrated on identifying the level of student’s ability in understanding probability concepts that has been included in the statistics course (STA150: Probability and Statistics 1). This course was enrolled by two groups of students (Group A and Group B) from the Faculty of Computer and Mathematical Sciences, Universiti Teknologi MARA Perak branch. Group A enrolled the course in December 2015 until March 2016 whilst Group B enrolled in June until November 2016 sessions. Since the aims of this study are to investigate the difference in students’ conceptual knowledge and understanding of probability concepts, as well as to examine which concepts were found most difficult by the students, hence the Rasch measurement approach was used to explore those aims. An instrument consists of 20 items in a test based on the “Counting Rule” topic was formed by an experienced lecturer to measure the level of student’s ability between the two groups. Based on the findings, it was found that there is a high reliability index of 0.93 (Group A) and 0.88 (Group B) which suggests the suitability of the instrument developed in this study to be replicated to the other samples, even though, the person student’s responses to the items for both groups appeared differently in terms of their difficulties in understanding the probability concepts which represents the student’s ability on this particular topic.

Examining the Psychometric Properties of a Modified Students’ Life Satisfaction Scale: A Rasch Measurement Approach

Drawing on data from the second wave of the Children’s Worlds: International Survey of Children’s Well-Being, this study examined the psychometric properties of a modified Students’ Life Satisfaction Scale (SLSS) using the Rasch measurement model in a sample of 1817 10- and 12-year-old children belonging to a highly collectivistic South Asian country, Nepal. Results of the study revealed that the modified SLSS data met the assumptions of the Rasch measurement model, and the internal structure of the modified SLSS represented a hierarchy of item difficulties. The modified SLSS items also exhibited adequate model-data fit, and they were found to be invariant across gender and age groups. Taken together, the modified SLSS is a psychometrically sound measure appropriate for assessing life satisfaction among Nepalese children.

Validation of a developed university placement test using classical test theory and Rasch measurement approach

University entrances examinations are conducted to ensure qualified applicant are placed into appropriate programs of their choices. The outcomes of the test have an important and significant value in taking appropriate decision on the applicant’s eligibility, the validity of that examination is paramount to achieving the set goal. The aim of this study is to provide empirical evidence of the construct validity of the newly developed Economics Test using traditional Classical Test Theory and Rasch Measurement Model. The developed Economics Test consists of 70 items after expert judgment and review was administered to 280 students, age 16-20 randomly selected from two public schools in Kano. The study employed a CTT and Rasch model to analyze the data using ITEMAN 4.3 and WINSTEPS 3.72.3 software. The softwares automatically generate the recommended estimate of the parameters to judge the quality of the test items. The results of CTT identified 17 problematic items using difficulty and discriminating index. The results of Rasch showed person statistics (Separation 2.40>2.00 and reliability 0.85>0.80) and item statistics (separation 3.73>3.0 and reliability 0.93>0.8) an excellent person and item reliability. The test measures unidimensional construct supported by the raw variance of 24.9% explained by measures. Investigation of the item person map revealed that the test covered a wide range of the examinees’ ability. Overall, using Rasch 10 misfitting construct irrelevant items were identified for deletion. While CTT provides information that is limited to two parameters, the Rasch results provide very detailed information on the quality of the test items. Thus both models can be integrated to generate enough evidence of validity and reliability items in the development of a standardize test.

Measuring students’ understanding in counting rules and its probability via e-learning mode: a Rasch measurement approach

Probability is a study of the rules that offers the foundational theory for the development of statistics. This sets out the investigation where students’ understanding of counting rules and its probability were explored using the Rasch measurement approach. A test instrument with 20 items was developed and administered to 74 students taking the STA150 Probability and Statistics course. Data were captured through an interactive e-learning platform that is dmodo.com and analyzed using Winsteps 3.81.0. The results from the Wright map showed that 83.8% of the students have the ability that matched well with the difficulty of the while 16.2% of the students need to be given more attention on the topic. The study was also able to show that the items can be replicated in other samples of similar characteristics. Keywords : students’ understanding; counting rules; probability Raschmeasurement model; Wright map.

Examining the psychometric properties of a sport-related concussion survey: a Rasch measurement approach

BackgroundAwareness of sport-related concussion (SRC) is an essential step in increasing the number of athletes or parents who report on SRC. This awareness is important, as there is no established data on medical care at youth-level sports and may be limited to individuals with only first aid training. In this circumstance, aside from the coach, it is the players and their parents who need to be aware of possible signs and symptoms. The aim of this study was to examine the psychometric properties of a parent and player concussion survey intended for use before and after an education campaign regarding SRC.Methods1441 questionnaires were received from parents and 284 questionnaires from players. The responses to the sixteen-item section of the questionnaire’s ‘recognition of signs and symptoms’ were submitted to psychometric analysis using the dichotomous and polytomous Rasch model via the Rasch Unidimensional Measurement Model software RUMM2030. The Rasch model of Modern Test Theory can be considered a refinement of, or advance on, traditional analyses of an instrument’s psychometric properties.ResultsThe main finding is that these sixteen items measure two factors: items that are symptoms of concussion and items that are not symptoms of concussion. Parents and athletes were able to identify most or all of the symptoms, but were not as good at distinguishing symptoms that are not symptoms of concussion. Analyzing these responses revealed differential item functioning for parents and athletes on non-symptom items. When the DIF was resolved a significant difference was found between parents and athletes.ConclusionsThe main finding is that the items measure two ‘dimensions’ in concussion symptom recognition. The first dimension consists of those items that are symptoms of concussion and the second dimension of those items that are not symptoms of concussion. Parents and players were able to identify most or all of the symptoms of concussion, so one would not expect to pick up any positive change on these items after an education campaign. Parents and players were not as good at distinguishing symptoms that are not symptoms of concussion. It is on these items that one may possibly expect improvement to manifest, so to evaluate the effectiveness of an education campaign it would pay to look for improvement in distinguishing symptoms that are not symptoms of concussion.

Using the 2006 PISA Questionaire to Evaluate the Measure of Educational Resources: A Rasch Measurement Approach

School educational resources are key when studying school improvement due to their influence on learning outcomes. Because of this, careful attention should be given to the way educational resources are operationalized and measured. Using the 2006 PISA American sample containing 166 schools, this study aims to validate the 13-item PISA School Educational Resource Scale with Rasch analysis. Winsteps software was used in the analysis and results were used to evaluate how well the instrument measured the construct of school educational resource. Findings revealed that the PISA 2006 data gave an overall indication of good fit to the model, despite the instrument not separating respondents well. In regards to the quality of the scale, the majority of items perform consistently with the model . However, for schools above the average educational resource threshold, it appears there is a need for more items to discriminate the situation.

Development and initial argument-based validation of a scoring rubric used in the assessment of L2 writing electronic portfolios

Abstract Although writing electronic portfolios (ePortfolios) help learners communicate digitally and provide a platform for works to be better collected and presented, challenges are present in their assessment. This paper reports the development and validation of a writing ePortfolio scoring rubric for an action research course for L2 learners. Using Bachman’s (2005) ‘Assessment Use Argument’ as a basis, two main claims to support the validity of the rubric were examined: (1) the rubric is a reliable tool; and, (2) the rubric is relevant to the construct being measured. A mixed-method approach was used in the development and validation of the eight-criteria analytic and holistic evaluation scoring rubric. Thirteen raters evaluated thirty-eight ePortfolios in the study. The analyses of raters’ ratings using a many-facet Rasch measurement approach and raters’ individual standardised open-ended interviews indicated that overall the rubric’s analytic categories functioned appropriately to assess the intended construct. However, the criterion ‘Ease of Navigation’ was found to be misfitting due to differences in raters’ evaluation of the same ePortfolios. Overall, the argument-based validation indicated that the scoring rubric is a reliable and valid instrument for the purpose of assessing L2 writing ePortfolios in the context for which it was developed.

Developing a composite indicator to measure civic participatory potential in two Chinese societies

AbstractThis study examined whether there is evidence in Hong Kong and Taiwan students’ early attitudes to civic participation that suggests they will adopt radical forms of civic participation and whether civic participatory potential of students from these two Chinese societies differed in early adolescence. To achieve these purposes, we used a Rasch measurement approach to construct comparable profiles. In doing so, we adopted conceptual and empirical approaches to construct a composite indicator and then tested validity and reliability of this indicator for the two societies, respectively. Such an approach is based on the assumption that reliable and accurate measurement is essential for theorizing the results of empirical studies. The data from the 2009 International Civics and Citizenship Education Study were also used for comparing the potential participation profiles. The results suggested that the unidimensional profile has good item fit and model fit for both societies, thereby valid and reliabl...

Estudio de confiabilidad y validez de la Escala de Manejo de Compromisos Identitarios de Utrecht, adaptada para medir procesos de formación de identidad de estudiantes en su universidad

The objective of this paper was to demonstrate the construct validity and predictive validity of the measurement framework presented by Crocetti, Rubini, and Meeus (2008), which was originally developed to measure the identity formation processes (such as achieving commitments) of individuals in various domains. An adapted version of the Utrecht-Management of Identity Commitments Scale (U-MICS) was used to measure university students’ identity formation processes, which are a part of students’ personal identities, at their university. The U-MICS seeks to measure three dimensions of identity formation: commitment, in-depth exploration, and reconsideration of commitment. A Rasch measurement approach was applied. The results showed that the multidimensional model validly represented students’ identity formation processes at university (construct validity), although some items need further improvement. Suggestions for improvement are discussed. The commitment and indepth exploration scales were positively related to students’ academic achievement, while the reconsideration of commitment scale was negatively related to the same (predictive validity).