Many-facet Rasch Model Research Articles

New Approaches in the Assessment of Pre-Service Teachers' Scientific Research Skills: Multidimensional-Many Facet Rasch Model Application

This survey study aimed to determine pre-service teachers’ skills in conducting and presenting scientific research process and to examine their peer-scoring behaviors. The participants consisted of 36 pre-service teachers and seven peer raters. The analytical rubric developed by the researchers and the performance task were used to collect data. The multidimensional many-facet Rasch measurement model was employed in data analysis. Rasch analysis was carried out with a fully crossed design. The analyses revealed that the least difficult criterion in the dimension of conducting the scientific research process was to perform data analysis while the most challenging criterion was to determine the research model and sample. In addition, the least difficult criterion in the dimension of making an oral presentation was to ensure the interest and participation of the audience, while the most challenging criterion was to make an effective start. The analysis of the rater facet demonstrated that the most severity rater was R3, while the most leniency rater was R4. The study argues that the multidimensional many-facet Rasch model can be used to present reliability and validity evidence in multidimensional performance evaluations.

Validating an integrated reading-into-writing scale with trained university students

Integrated tasks are often used in higher education (HE) for diagnostic purposes, with increasing popularity in lingua franca contexts, such as German HE, where English-medium courses are gaining ground. In this context, we report the validation of a new rating scale for assessing reading-into-writing tasks. To examine scoring validity, we employed Weir’s (2005) socio-cognitive framework in an explanatory mixed-methods design. We collected 679 integrated performances in four summary and opinion tasks, which were rated by six trained student raters. They are to become writing tutors for first-year students. We utilized a many-facet Rasch model to investigate rater severity, reliability, consistency, and scale functioning. Using thematic analysis, we analyzed think-aloud protocols, retrospective and focus group interviews with the raters. Findings showed that the rating scale overall functions as intended and is perceived by the raters as valid operationalization of the integrated construct. FACETS analyses revealed reasonable reliabilities, yet exposed local issues with certain criteria and band levels. This is corroborated by the challenges reported by the raters, which they mainly attributed to the complexities inherent in such an assessment. Applying Weir’s (2005) framework in a mixed-methods approach facilitated the interpretation of the quantitative findings and yielded insights into potential validity threads.

Assessing the Psychometric Qualities of the Data-Informed School Leadership Survey

Purpose: There is little consensus in the literature regarding a) what it means for a school leader to lead with data, and b) how to measure data-informed leadership in a reliable and valid way. This study examines the psychometric properties of an operational measure intended to assess the extent to which a school leader is a ‘data-informed school leader. The measurement invariance, reliabilities and construct and predictive validities of the Data-Informed School Leadership Survey (DISL Survey) are assessed using various psychometric statistical techniques. Methods: Using data collected from 155 teachers from 7 public schools in a southern state, the following psychometric statistics used to address our purpose: the Many-Facet Rasch (MFR) Model, Bayesian second-order Confirmatory Factor Analysis (CFA), Bayesian Structural Equation Modeling- Multiple Indicators, Multiple Causes analysis (Bayesian SEM-MIMIC), and reliability analysis. Findings: Results show an adequate fit from all MFR, Bayesian CFA, and MIMIC models and a high reliability (Cronbach α = .98). The DISL Survey instrument exhibits sound psychometric properties. Results likewise confirm the value of using MFR modeling and Bayesian methods to examine the psychometric properties of DISL Survey as a means of improving educational leadership measures. Implications for Research and Practice: Data from this study confirm the validity and reliability of the Data-Informed School Leadership Survey (DISL Survey) as an instrument to assess the strengths and weaknesses of Data-Informed School Leadership (DISL) and as a means for providing feedback for improving such leadership. Heretofore a measure for assessing this leadership was non-existent.

Reexamining subjective creativity assessments in science tasks: An application of the rater-mediated assessment framework and many-facet Rasch model.

Reexamining subjective creativity assessments in science tasks: An application of the rater-mediated assessment framework and many-facet Rasch model.

Do source use features impact raters’ judgment of argumentation? An experimental study

This experimental study explores how source use features impact raters’ judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were adapted from essays written by EPT test-takers. These responses were crafted to reflect different conditions of source use features, namely source use quantity and quality. Rater scores were analyzed using the many-facet Rasch model and mixed two-way analyses of variance (ANOVAs) to examine how they are affected by source use features and rater experience. Results show that source use features impacted the argumentation scores assigned by raters. Paragraphs with more source text ideas that are better incorporated received the highest argumentation scores, and vice versa for those with limited, poorly integrated source information. Rater experience impacted scores but did not influence rater performance meaningfully. The findings of this study connect specific source use features with raters’ evaluation of argumentation, helping to further disentangle the relationships among examinee performance, rater decision, and task features of integrated argumentative writing tests. They also provide meaningful implications for writing assessment research and practices.

A Comparison of Psychometric Properties of the American Board of Anesthesiology’s In-Person and Virtual Standardized Oral Examinations

Abstract Purpose The COVID-19 pandemic prompted training institutions and national credentialing organizations to administer examinations virtually. This study compared task difficulty, examiner grading, candidate performance, and other psychometric properties between in-person and virtual standardized oral examinations (SOEs) administered by the American Board of Anesthesiology. Method This retrospective study included SOEs administered in person from March 2018 through March 2020 and virtually from December 2020 through November 2021. The in-person and virtual SOEs share the same structure, including 4 tasks of preoperative evaluation, intraoperative management, postoperative care, and additional topics. The Many-Facet Rasch Model was used to estimate candidate performance, examiner grading severity, and task difficulty for the in-person and virtual SOEs separately; the virtual SOE was equated to the in-person SOE by common examiners and all tasks. The independent-samples and partially overlapping–samples t tests were used to compare candidate performance and examiner grading severity between these 2 formats, respectively. Results In-person (n = 3,462) and virtual (n = 2,959) first-time candidates were comparable in age, sex, race and ethnicity, and whether they were U.S. medical school graduates. The mean (standard deviation [SD]) candidate performance was 2.96 (1.76) logits for the virtual SOE, which was statistically significantly better than that for the in-person SOE (mean [SD], 2.86 [1.75]; Welch independent-samples t test, P = .02); however, the effect size was negligible (Cohen d = 0.06). The difference in the grading severity of examiners who rated the in-person (n = 398; mean [SD], 0.00 [0.73]) vs virtual (n = 341; mean [SD], 0.07 [0.77]) SOE was not statistically significant (Welch partially overlapping–samples t test, P = .07). Conclusions Candidate performance and examiner grading severity were comparable between the in-person and virtual SOEs, supporting the reliability and validity of the virtual oral exam in this large-volume, high-stakes setting.

The 21st-century skills scales: many facet Rasch measurements

This study aims to develop a 21st century skills scale for prospective school counselor. The method used is the research and development. The data were analyzed using content analysis/expert judgment with many-facet Rasch model (MFRM). Three facets were employed: scale, item, and measurement criteria. The initial version of the scale comprises 129 items measuring four aspects: way of thinking, way of working, tools for working, and living in the world. However, 78 items were deemed valid following the expert judgment analysis result. The inter-rater agreement opportunity shows raters’ similarity score of 10,836, implying that experts in this study tend to give similar assessments. This research results in a 21st century skills scale for prospective school counselors with a total of 78 valid items. The purpose of this study was to develop a 21st century skills scale for prospective school counselors that had been fulfilled, the instrument had been tested for validity and reliability based on expert judgment. The current research recommends that further research be carried out to develop a 21st century skills scale instrument for prospective educators in other scientific fields.

THE MANY-FACET RASCH MODEL IN THE ANALYSIS OF INSTRUMENTS MEASURING TEACHERS' ATTITUDES TOWARDS EARTHQUAKE DISASTER PREPAREDNESS IN INDONESIA

Materials and Methods: This study employed a survey research design with an exploratory quantitative approach using Many-Facet Rasch Model (MFRN) analysis. The data used for the analysis of the assessment instrument measuring teachers' attitudes towards earthquake disaster preparedness in schools were field data obtained from a survey conducted by three raters assessing 33 elementary school teachers in Palu City. The instrument consisted of 20 items, resulting in a total of 1980 collected data points. Results: The item reliability value was 0.94, indicating that the instrument's quality falls within the "excellent" criteria with a high level of reliability. However, the analysis results indicated that some items assessing teachers' attitudes towards earthquake disaster preparedness in schools had a high level of difficulty. This suggests the need for further examination of disaster preparedness topics in specific areas that teachers may not be adequately prepared for. There was a significant variation in teachers' abilities to answer the instrument's items, highlighting the necessity for a more differentiated approach. Teachers with lower abilities should receive additional assistance and support to ensure that they can also enhance their preparedness.

Expert judgement of collaborative cloud classroom quality and its criteria using the many-facets rasch model

Collaborative Cloud Classroom (3CR) is an educational application that provides flexibility in the learning process. This specialized tool is designed to improve analytical and teambuilding skills, as well as enable students to document ideas, which are subsequently developed into bubble diagrams and text outlines. This study aims to assess and analyze the menu qualities possessed by the 3CR application using the Many-Facet Rasch Model (MFRM) analysis using 5 raters and 6 menus. A rating assessment was applied to 5 criteria, namely usability, functionality, visual communication, learning design, and security. In this case, a total of 135 from 150 data points were coded into FACETS software. The results showed that the six menus in the 3CR application were valid and reliable, as well as met the quality standards of learning tools. Security criteria were also the most difficult standards for raters to measure in all the application menus. The study also found that Rater A was more severe, and Rater E was more lenient in rating the 3CR menu. Rater B has overfitting and detected bias between rater gender and 3CR's menu (Discussion Forum and Learning Reflection menu) based on MFRM analysis.

A comparison of the American Board of Anesthesiology's in-person and virtual objective structured clinical examinations

BackgroundThe American Board of Anesthesiology's Objective Structured Clinical Examination (OSCE), as a component of its initial certification process, had been administered in-person in a dedicated assessment center since its launch in 2018 until March 2020. Due to the COVID-19 pandemic, a virtual format of the exam was piloted in December 2020 and was administered in 2021. This study aimed to compare candidate performance, examiner grading severity, and scenario difficulty between these two formats of the OSCE. MethodsThe Many-Facet Rasch Model was utilized to estimate candidate performance, examiner grading severity, and scenario difficulty for the in-person and virtual OSCEs separately. The virtual OSCE was equated to the in-person OSCE by common examiners and common scenarios. Independent-samples t-test was used to compare candidate performance, and partially overlapping samples t-tests were applied to compare examiner grading severity and scenario difficulty between the in-person and virtual OSCEs. ResultsThe in-person (n = 3235) and virtual (n = 2934) first-time candidates were comparable in age, sex, race/ethnicity, and whether U.S. medical school graduates. The virtual scenarios (n = 35, mean [0.21] ± SD [0.38] in logits) were more difficult than the in-person scenarios (n = 93, 0.00 ± 0.69, Welch's partially overlapping samples t-test, p = 0.01); there were no statistically significant differences in examiner severity (n = 390, −0.01 ± 0.82 vs. n = 304, −0.02 ± 0.93, Welch's partially overlapping samples t-test, p = 0.81) or candidate performance (2.19 ± 0.93 vs. 2.18 ± 0.92, Welch's independent samples t-test, p = 0.83) between the in-person and virtual OSCEs. ConclusionsOur retrospective analyses of first-time OSCEs found comparable candidate performance and examiner grading severity between the in-person and virtual formats, despite the virtual scenarios being more difficult than the in-person scenarios. These results provided assurance that the virtual OSCE functioned reasonably well in a high-stakes setting.

The Role of Time on Performance Assessment (Self, Peer and Teacher) in Higher Education: Rater Drift

This study aimed to investigate the change in teacher candidates' oral presentation skills over time through self, peer, and teacher assessments using the rater drift method. A longitudinal descriptive research model was used as a quantitative research approach to achieve this aim. The study group consisted of 47 teacher candidates receiving formation education at a state university in the Eastern Anatolia Region and an instructor teaching the course. An analytical rubric was used as a data collection tool to evaluate the candidates' oral presentation skills. The data collection process lasted six weeks in total. Since the performance evaluation process aimed to examine the change over time, the many-facet Rasch model was used. When the findings of the study were examined, it was determined that the rater behavior of teacher candidates had statistically significant differences at the group level over time. It was found that 26 out of 48 peer raters had rater drift in their evaluations. It was also found that the majority of rater drift over time was positive, meaning that evaluators became more generous over time. Another result obtained in the study was that teacher assessment did not show rater drift over time, with similar ratings for six weeks. The study’s findings were discussed with previous studies in the literature, and recommendations were made to researchers.

Measurement of Students' Chemistry Practicum Skills Using Many Facets Rash Model

<p style="text-align:justify">The accuracy of assessing the capabilities of the process and product in chemical practice activities requires appropriate measurement procedures to be followed. It is crucial to identify the components that can introduce bias while measuring student abilities during the measurement process. This study aims to identify the components or criteria used by teachers to assess student performance in practicum activities and analyze the quality of the rubrics developed. The study was conducted with the participation of three raters, 27 high school students, and nine assessment criteria. A quantitative descriptive approach was employed using the many-facet Rasch model (MFRM) analysis for measurement. The results of the MFRM analysis show no significant measurement bias, with data measurement facets fitting the MFRM model. The reliability of all the facets meets the criteria, and the scale predictor functions appropriately. While all students can easily pass four out of nine items, five items can only be partially passed by students. The assessment criteria that require special attention include communication skills, tools and assembly, interpretation, cleanliness, and accuracy when performing practicums. These criteria provide feedback for teachers and students to ensure successful practicum activities. The Discussion section of this study delves into the findings and their implications.</p>

Evidence supporting performance measures of laparoscopic appendectomy through a novel surgical proficiency assessment tool and low-cost laparoscopic training system.

Laparoscopic training remains inaccessible for surgeons in low- and middle-income countries, limiting its widespread adoption. We developed a novel tool for assessment of laparoscopic appendectomy skills through ALL-SAFE, a low-cost laparoscopy training system. This pilot study in Ethiopia, Cameroon, and the USA assessed appendectomy skills using the ALL-SAFE training system. Performance measures were captured using the ALL-SAFE verification of proficiency tool (APPY-VOP), consisting of a checklist, modified Objective Structured Assessment of Technical Skills (m-OSATS), and final rating. Twenty participants, including novice (n = 11), intermediate (n = 8), and expert (n = 1), completed an online module covering appendicitis management and psychomotor skills in laparoscopic appendectomy. After viewing an expert skills demonstration video, participants recorded their performance within ALL-SAFE. Using the APPY-VOP, participants rated their own and three peer videos. We used the Kruskal-Wallis test and a Many-Facet Rasch Model to evaluate (i) capacity of APPY-VOP to differentiate performance levels, (ii) correlation among three APPY-VOP components, and (iii) rating differences across groups. Checklist scores increased from novice (M = 21.02) to intermediate (M = 23.64) and expert (M = 28.25), with differentiation between experts and novices, P = 0.005. All five m-OSATS domains and global summed, total summed, and final rating discriminated across all performance levels (P < 0.001). APPY-VOP final ratings adequately discriminated Competent (M = 2.0), Borderline (N = 1.8), and Not Competent (M = 1.4) performances, Χ2 (2,85) = 32.3, P = 0.001. There was a positive correlation between ALL-SAFE checklist and m-OSATS summed scores, r(83) = 0.63, P < 0.001. Comparison of ratings suggested no differences across expertise levels (P = 0.69) or location (P = 0.66). APPY-VOP effectively discriminated between novice and expert performance in laparoscopic appendectomy skills in a simulated setting. Scoring alignment across raters suggests consistent evaluation, independent of expertise. These results support the use of APPY-VOP among all skill levels inside a peer rating system. Future studies will focus on correlating proficiency to clinical practice and scaling ALL-SAFE to other settings.

Psychometric properties of the successful school leadership survey

PurposeThis study extends research on one of the most frequently cited school leadership frameworks by examining the psychometric properties of the instrument designed to assess many of the practices included in that framework.Design/methodology/approachUsing data collected from 1,401 teachers the study examined the instrument’s measurement invariance, score reliabilities, as well as construct and predictive validities. Polytomous latent trait models (Many-Facet Rasch model), scale and principal component analysis using second-order Confirmatory Factor Analysis, and Structural Equation Modeling (SEM)-Path modelling were used for these purposes.FindingsFindings report levels of score reliability and valid score inferences. Results concerning the predictive validity of the instrument indicate a complex set of relations among the domains of leadership practices measured by the instrument, variables selected as mediators of leaders’ influence, and their direct and indirect effects on student learning.Research limitations/implicationsThis study provides researchers with a reliable and valid instrument for use in their future research. Data for the study were provided by elementary teachers in one US state. The extent to which results of the instrument are valid across different cultural and organizational settings remains to be determined.Practical implicationsLeadership developers may find the instrument useful for assessing the strengths and weaknesses of those participating in their programs while leaders themselves many find the instrument useful for self-diagnosis.Originality/valueThis study contributes to the development of school leadership measures by including Rasch modeling among the methods used for examining the instrument’s psychometric properties.

Item and rater variabilities in students’ evaluation of teaching in a university in Ghana: application of Many-Facet Rasch Model

This research examined the item and rater variabilities in students' evaluation of teaching and courses exercise at the University of Cape Coast (UCC) through the lenses of the Many-Facet Rasch Model (MFRM). The study covered students during the 2019/2020 academic year in the selected university, analysing secondary data obtained from the Directorate of Academic Planning and Quality Assurance, UCC (DAPQA-UCC). The data were analysed by conducting partial credit MFRM analyses. It was found that the sources of measurement errors in the student evaluation exercise included halo effect, non-functional item structure, inconsistent students' ratings, rater leniency, and non-functional rating scale. It was concluded that data from students' appraisal of lecturers’ teaching should be used with caution. It was recommended that DAPQA-UCC and the university management should train students on the evaluation of teaching, as well as review the existing evaluation form for appraising courses and teaching by subjecting the instrument to rigorous validation procedures.

Examination of map reading skills with orienteering activity: An example of Many Facet Rasch Model

The purpose of the current study is to examine the map reading skills of Social Studies pre-service teachers with orienteering, which is an activity-based and more active practice. To this end, a total of 10 students attending the Department of Social Studies Teaching in the Education Faculty of Burdur Mehmet Akif Ersoy University and taking the course of Map Skills and Applications were selected. An analytical rubric consisting of four criteria and scored in four categories was used to collect data in the study. The content validity of the developed rubric was calculated with the Davis Technique and it was thought that sufficient evidence was obtained for the content validity. During the orienteering activity, the map reading skills of the students were scored by 5 raters with this rubric in terms of four criteria, direction/location, recognizing signs/symbols, using landforms and managing time. They were examined with the many-facet Rasch model (MFRM). Map reading skills were evaluated according to the severity/leniency of the raters and the difficulty of the students in exhibiting the behavior. The results of the analysis showed that the agreement between the raters was found to be good. It was also concluded that the most difficult skill is determining direction/location and the easiest skill is using landforms.

Raters’ Assessment Quality in Measuring Teachers’ Competency in Classroom Assessment: Application of Many Facet Rasch Model

This study examines the raters’ assessment quality when measuring teachers’ competency in Classroom Assessment (CA) using the Many Facet Rasch Model (MFRM) analysis. The instrument used consists of 56 items built based on 3 main constructs: knowledge in CA, skills in CA, and attitude towards CA. The research design of this study is a quantitative method with a multi-rater approach using a questionnaire distributed to the raters. Respondents are 68 raters consisting of The Head of Mathematics and Science Department, The Head of Mathematics Panel, and the Mathematics Teacher to assess 27 ratees. The ratees involved in this study are 27 secondary school Mathematics teachers from Selangor. The results show that among the advantages of MFRM are that it can determine the severity and consistency level of the raters, also detect bias interaction between rater and ratee. Although all raters were given the same instrument, the same aspects of evaluation, and scale category, MFRM can compare the severity level for each rater individually. Furthermore, MFRM can detect measurement biases and make it easier for researchers to communicate about the research findings. MFRM has the advantage of providing complete information and contributes the understanding of the consistency analysis of the rater’s judgement with quantitative evidence support. This indicates that MFRM is an alternative model suitable to overcome the limitations in Classical Test Theory (CTT) statistical models in terms of multi-rater analysis.

A Bayesian many-facet Rasch model with Markov modeling for rater severity drift.

Fair performance assessment requires consideration of the effects of rater severity on scoring. The many-facet Rasch model (MFRM), an item response theory model that incorporates rater severity parameters, has been widely used for this purpose. Although a typical MFRM assumes that rater severity does not change during the rating process, in actuality rater severity is known to change over time, a phenomenon called rater severity drift. To investigate this drift, several extensions of the MFRM have been proposed that incorporate time-specific rater severity parameters. However, these previous models estimate the severity parameters under the assumption of temporal independence. This introduces inefficiency into the parameter estimation because severities between adjacent time points tend to have temporal dependency in practice. To resolve this problem, we propose a Bayesian extension of the MFRM that incorporates time dependency for the rater severity parameters, based on a Markov modeling approach. The proposed model can improve the estimation accuracy of the time-specific rater severity parameters, resulting in improved estimation accuracy for the other rater parameters and for model fitting. We demonstrate the effectiveness of the proposed model through simulation experiments and application to actual data.

Assessment as Learning: How Does Peer Assessment Function in Students' Learning?

Peer assessment (PA) is employed as one fundamental practice of classroom-based assessment in terms of its learning-oriented and formative nature. The exercise of peer assessment has multiple and additional benefits for student learning. However, research into the learning processes in peer assessment is scarce both in theory and in practice, making it difficult to evaluate and pinpoint its value as a tool in assessment as learning (AaL). This study focuses both on the learning process and outcome through assessment activities. We set out with three goals in mind: (1) to examine students' assessment performance in context, (2) to evaluate its impact on student progress, and (3) to illuminate teachers on organizing assessment activities. Three specific research questions are answered in this study: (1) How do student raters perform in the process of PA in an advanced English class? (2) To what extent do assessment activities influence the students' speaking ability? (3) What are students' perceptions of PA regarding its benefits and caveats? A total of 29 undergraduate students participated in two assessment activities on argumentative speaking. Many-Facet Rasch Model analysis was conducted to measure the rater effects both at the group level and the individual level. Bias/interaction analyses were performed to diagnose rater behavior in different contexts including the rating session, speaking session, and peer assessment vs. self-assessment. Questionnaire and semi-structured interview data were also collected to explore factors and strategies that could interfere with PA as AaL. Results show that students exhibited stable rating behavior and made progress in argumentative speaking in all dimensions, including delivery, organization, and language use. They are more stringent with themselves than with peers although there is one rare case with a bias against peers. Participants acknowledged the benefits of PA but also shared reasonable concerns in practice. This study validated the feasibility and the effectiveness of PA for student learning. Discussion on findings and guidelines for effective implementation of PA as AaL are provided.

Reliability of Ratings of Multidimensional Fluency Scale with Many-Facet Rasch Model

The aim of this study was to determine whether the reliability of raters was provided by assessing reading prosody using the Multidimensional Fluency Scale (MDFS). The study was completed with a cross-sectional design, and in line with this, the prosodic reading skills of 41 fifth-grade students were rated by elementary school classroom teachers and Turkish language arts teachers using the MDFS. Data obtained from the ratings were analyzed with the many-facet Rasch model (MFRM). When the findings are investigated, the reading prosody rubric used in the research served the purposes of the reading prosody criteria, the sub-dimensions of the rubric could be reliably differentiated, the determined criteria were reliable, and the criteria categories appear to be adequate. Additionally, the severity and leniency of raters were found to differ, and Turkish language arts teachers were found to perform more severe ratings than classroom teachers. It was found that raters were ranked reliably in terms of severity/leniency, and that their levels of severity/leniency differed from each other. Another result obtained is that the prosody criterion that students completed with the most difficulty was phrasing. Therefore, it was concluded that the MDFS is a reliable rubric and that researchers and teachers can reliably use it to assess prosodic reading skills.