Many-facet Rasch Measurement Research Articles

New Approaches in the Assessment of Pre-Service Teachers' Scientific Research Skills: Multidimensional-Many Facet Rasch Model Application

This survey study aimed to determine pre-service teachers’ skills in conducting and presenting scientific research process and to examine their peer-scoring behaviors. The participants consisted of 36 pre-service teachers and seven peer raters. The analytical rubric developed by the researchers and the performance task were used to collect data. The multidimensional many-facet Rasch measurement model was employed in data analysis. Rasch analysis was carried out with a fully crossed design. The analyses revealed that the least difficult criterion in the dimension of conducting the scientific research process was to perform data analysis while the most challenging criterion was to determine the research model and sample. In addition, the least difficult criterion in the dimension of making an oral presentation was to ensure the interest and participation of the audience, while the most challenging criterion was to make an effective start. The analysis of the rater facet demonstrated that the most severity rater was R3, while the most leniency rater was R4. The study argues that the multidimensional many-facet Rasch model can be used to present reliability and validity evidence in multidimensional performance evaluations.

The equivalence of a high-stakes objective structured clinical exam adapted to suit a virtual delivery format.

The COVID-19 pandemic necessitated rapid adaptation of clinical competence assessments, including the transition of Objective Structured Clinical Examinations (OSCE) from in-person to virtual formats. This study investigates the construct equivalence of a high-stakes OSCE, originally designed for in-person delivery, when adapted for a virtual format. A retrospective analysis was conducted using OSCE scores from the Internationally Educated Nurse Competency Assessment Program (IENCAP®). Data were collected from 15 exam administrations between January 2018 and June 2022, encompassing 2021 examinees (1936 in-person, 85 virtual). The Many-Facet Rasch Measurement (MFRM) model was employed to analyze the invariance of examinee ability, case difficulty, and criteria difficulty across in-person and virtual formats. Results revealed overall examinee ability estimates remained invariant regardless of the OSCE format, while invariant violations were identified in only three of the 15 cases (N = 20%) adapted to suit the virtual format. The most significant adaptation, namely the use of a verbal physical examination to suit the virtual context achieved equivalence to its hands-on in-person counterpart given evidence of invariance across criteria estimates. Interestingly, criteria scores in invariant violated cases displayed a higher level of stability or consistency across the virtual OSCE formats versus their in-person counterpart highlighting a potential benefit of the virtual versus in-person format and potentially linked to the verbal physical examination. The study found that while examinee ability and case difficulty estimates exhibited some invariance between in-person and virtual OSCE formats, criteria involving physical assessments faced challenges in maintaining construct equivalence. These findings highlight the need for careful consideration in adapting high-stakes clinical assessments to virtual formats to ensure fairness and reliability.

Assessing interactional competence through group discussion: A mixed methods validation

The purpose of this mixed methods study was to assess the validity and functionality of an analytic rating scale for the assessment of interactional competence (IC). The participants were 79 low- to high-proficiency Japanese university students who completed 10-minute small-group discussions. Video recordings of the discussions were assessed by raters using the rating scale. The rater scores were then analyzed using many-facet Rasch measurement (MFRM) which indicated a very good fit to the model. The data were subsequently analyzed using generalizability theory in the form of a G-study and d-study. Those studies showed that the rating scale could be used with fewer raters, therefore increasing practicality without a substantial decrease in reliability. In addition to quantitative data, qualitative data were also collected in the form of interviews with raters and comments they made during assessment. Several raters noted unexpected participant behaviors which were difficult to evaluate using the rating scale, as well as ambiguous language in some category descriptors. The qualitative data provided an invaluable supplement to the quantitative analyses which did not indicate major issues with the rubric. Both forms of data were used to revise the original rating scale and those changes are discussed. This study adds to the limited but growing number of mixed methods studies on IC assessment.

A many-facet Rasch measurement model approach to investigating objective structured clinical examination item parameter drift.

Objective Structured Clinical Examinations (OSCEs) are widely used for assessing clinical competence, especially in high-stakes environments such as medical licensure. However, the reuse of OSCE cases across multiple administrations raises concerns about parameter stability, known as item parameter drift (IPD). AIMS &OBJECTIVES: This study aims to investigate IPD in reused OSCE cases while accounting for examiner scoring effects using a Many-facet Rasch Measurement (MFRM) model. Data from 12 OSCE cases, reused over seven administrations of the Internationally Educated Nurse Competency Assessment Program (IENCAP), were analyzed using the MFRM model. Each case was treated as an item, and examiner scoring effects were accounted for in the analysis. The results indicated that despite accounting for examiner effects, all cases exhibited some level of IPD, with an average absolute IPD of 0.21 logits. Three cases showed positive directional trends. IPD significantly affected score decisions in 1.19% of estimates, at an invariance violation of 0.58 logits. These findings suggest that while OSCE cases demonstrate sufficient stability for reuse, continuous monitoring is essential to ensure the accuracy of score interpretations and decisions. The study provides an objective threshold for detecting concerning levels of IPD and underscores the importance of addressing examiner scoring effects in OSCE assessments. The MFRM model offers a robust framework for tracking and mitigating IPD, contributing to the validity and reliability of OSCEs in evaluating clinical competence.

Developing a Rubric for Interactional Competence Using Many-Facet Rasch Measurement

The teaching and assessment of interactive speaking skills is a key aim in many English-language programs, and the right assessment rubric is a key component of any effective course. There is no one-size-fits-all approach and rubrics need to be validated, feedback collected, and revisions made. This paper reports on a small-scale, exploratory study undertaken to develop and improve a rubric for assessing interactive discussion skills. Utilizing many-facet Rasch measurement (MFRM), the paper reports on an analysis of the rubric originally used on the course. Based on these findings, the rubric was revised and subject to a second round of analysis. The findings indicate that the revisions led to considerably improved rubric function, due primarily to a reduced number of scale points and more clearly defined rubric categories. Finally, the paper suggests a number of recommendations for rubric creation that can be applied to a range of assessment contexts. 概要インタラクティブなスピーキングスキルの指導と評価は、多くの英語学習プログラムにおいて重要な目標であり、適切な評価ルーブリックは効果的な授業の重要な要素である。万能なアプローチは存在しない為、フィードバックを収集し、ルーブリックは検証、改善される必要がある。本稿では、対話型ディスカッションのスキルを評価するためのルーブリックを開発・改善するために行われた、小規模で探索的な研究について考察する。多相ラッシュ測定（MFRM）を活用し、当初コースで使用されていたルーブリックの分析について報告する。その結果に基づき、ルーブリックが改訂され2回目の分析が行われたところ、ルーブリックの機能に大幅な改善が見られた。その主な理由は、尺度点数を減らし、ルーブリックのカテゴリーをより明確に定義したことにあることがわかった。最後に、本論文は様々な評価の文脈に適用可能なルーブリック作成に関する多くの提言を提示している。

Detecting Randomness Effect among Raters in Physics Essay Items using Many-Facet Rasch Measurement

This study sought to detect randomness effects among raters in physics essay items using Many-Facet Rasch Measurement. The research design adopted for this study is descriptive research design based on survey method. The population of the study comprised eighty-eight (88) public schools in all the local government areas with a physics student population of 3,642 students and ninety-four (94) physics teachers in all the Senior Secondary Schools in Uyo Senatorial District for the 2022/2023 academic session. Three hundred and sixty-four (364) SSS3 physics students and 37 physics teachers from the 31 selected secondary schools in Uyo Senatorial District were sampled using multistage sampling technique for effective selection. The multi-stage sampling technique was adopted for the study. The instrument used for data collection was Physics Achievement Test (PAT) obtained from WAEC and NECO 2020 Physics Essay items with reliability coefficients of 0.91 and 0.90 respectively. The finding revealed that most of the individual raters commit randomness effect when rating physics essay Items. It also revealed there is a significant difference at the rater's group level exhibiting randomness effect when rating physics essay items, which implies that there is no group-level randomness effect present among raters when rating physics essay items. We concluded that rater effects are sources of variance in performance ratings that are associated with the raters' behaviour and not the actual performance of the ratee. It was recommended that raters should follow the rating guidelines to reduce the impact of randomness in their ratings and provide more accurate and objective evaluations.

Applicability of Fine-Tuned ChatGPT to Automated Essay Scoring

ChatGPT, released in 2022, has garnered attention due to its adaptability through prompt engineering, enabling users to guide its responses. It is important to note that the extent to which users can modify ChatGPT remains limited, as its core embeddings stay unaltered through prompt engineering. Thus, this study aims to evaluate the effectiveness of ChatGPT and its fine-tuned model in essay evaluation compared to human raters. A total of 904 essays from the YELC 2011, on the subject of physical punishment, were selected for this study. Among these, 723 essays were used for fine-tuning ChatGPT, and the remaining 181 were reserved for testing the language model. Additionally, an extra set of 200 essays on different topics, such as driving and medical issues, was included to evaluate the language model’s performance across various themes. Inter-rater reliability indices, including measures like correlation, agreement, Cohen’s kappa, and Krippendorff’s alpha, along with many-facet Rasch measurement analysis, collectively indicated that the current version of ChatGPT (gpt-3.5-turbo-0613) is not yet poised to fully supplant human raters in essay scoring. Nevertheless, through the fine-tuning process, the model demonstrated a significant level of agreement with human raters and exhibited a marked degree of consistency.

Making each point count: Revising a local adaptation of the Jacobs et al.’s (1981) ESL COMPOSITION PROFILE rubric

In this Brief Report, we describe an evaluation of and revisions to a rubric adapted from the Jacobs et al.’s (1981) ESL COMPOSITION PROFILE, with four rubric categories and 20-point rating scales, in the context of an intensive English program writing placement test. Analysis of 4 years of rating data (2016–2021, including 434 essays) using many-facet Rasch measurement demonstrated that the 20-point rating scales of the Jacobs et al. rubric functioned poorly due to (a) questionably small distinctions in writing quality between successive score categories and (b) the presence of several disordered categories. We reanalyzed the score data after collapsing the 20-point scales into 4-point scales to simulate a revision to the rubric. This reanalysis appeared promising, with well-ordered and distinct score categories, and only a trivial decrease in person separation reliability. After implementing this revision to the rubric, we examined data from recent administrations (2022–2023, including 93 essays) to evaluate scale functioning. As in the simulation, scale categories were well-ordered and distinct in operational rating. Moreover, no raters demonstrated exceedingly poor fit using the revised rubric. Findings hold implications for other programs adopting/adapting the PROFILE or a similar rubric.

Topic Familiarity Effects on Performance in Speaking Assessment Tasks

Speaking assessment is believed to be difficult in its expansion and execution. Thus, it is a challenge for teachers in preparing students for speaking tests. This study’s purpose was to identify whether topic familiarity affects speaking performance among students who were preparing to sit for the high-stakes Malaysian University English Test speaking test. The study aimed to investigate the validity and reliability of the speaking assessment measures and, subsequently, to examine the speaking tests’ item difficulty measures differences according to topic familiarity level. Data were collected from 40 non-native speakers of English among Malaysian Form Six pre-university students who were preparing for their MUET test. The researcher conducted two practice speaking tests, which used retired papers of the MUET CEFR-aligned speaking tests, to the 40 participants who were grouped into 10 speaking test groups. The practice speaking tests were video and audio-recorded. Topic familiarity was measured using self-report questionnaires. In the second phase, each of the seven appointed raters scored all 40 students’ responses in two speaking tests consisting of two speaking tasks assessed across four criteria: task fulfillment, language, communicative ability, and group discussion. Many-facet Rasch measurement results revealed significant differences translating to significant influence of topic familiarity on speaking performance. The present study’s results not only confirmed the significance of topic familiarity in the preparation for speaking assessments, but also highlighted the need for formal teaching on potential topic themes that are commonly encountered in such assessments, particularly those that are at the post-secondary level. These findings imply importance in designing speaking tests taking into account test-taker topic familiarity.

Triangulating natural language processing (NLP)-based analysis of rater comments and many-facet Rasch measurement (MFRM): An innovative approach to investigating raters’ application of rating scales in writing assessment

Rater comments tend to be qualitatively analyzed to indicate raters’ application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The data consisted of ratings on 987 essays by 36 raters (a total of 3948 analytic scores and 1974 rater comments) on a post-admission English Placement Test (EPT) at a large US university. We computed a set of comment-based features based on the analytic components and evaluative language the raters used to infer whether raters were aligned to the scale. For data triangulation, we performed correlation analyses between the MFRM measures of rater performance and the comment-based measures. Although the EPT raters showed overall satisfactory performance, we found meaningful associations between rater comments and performance features. In particular, raters with higher precision and fit to what the Rasch model predicts used more analytic components and used evaluative language more similar to the scale descriptors. These findings suggest that NLP techniques have the potential to help language testers analyze rater comments and understand rater behavior.

SPEAKING ENGLISH PERFORMANCE ASSESSMENT WITH THE FACET RASCH MEASUREMENT MODEL

This study aims to assess students' English-speaking abilities based on peer assessment. This study is a quantitative study involving 10 students. Data was collected using tests and student speaking assessment rubrics with score criteria from 1 to 5. Speaking assessment criteria are pronunciation, grammar, vocabulary, fluency and understanding. Data were analyzed using Many Faceted Rasch Measurement (MFRM). The Facets Rasch Measurement model is able to see the interaction between respondents and items at once. The research results show that the item index for criteria/quality (6.39), speaker (0.51), and rater (5.32) as well as the standard deviation value clearly shows a good distribution of item difficulty. Criterion reliability is 0.98 for raters is 0.21, for raters is 0.97.

Investigating The Central Tendency Effect Among Raters in Complex Problem-Solving Skills in Physics Essay Items: An Application of Many-Facet Rasch Measurement

The study sought to investigate the central tendency effect among raters in complex problem-solving skills in physics essay items using Many-Facet Rasch Measurement. This became necessary because the central tendency effect can affect students' scores. The research design adopted for this study is a descriptive research design based on the survey method. The population of the study comprised eighty-eight (88) public schools in all the local government areas with a physics student population of 3642 students and ninety-four (94) physics teachers in all the Senior Secondary Schools in Uyo Senatorial District for the 2022/2023 academic session. 364 SS3 physics students and 37 physics teachers from the 31 selected secondary schools in Uyo Senatorial District were sampled using a multistage sampling technique for effective selection. The multi-stage sampling technique was adopted for the study. The instrument used for data collection was the Physics Achievement Test (PAT) obtained from WAEC and NECO 2020 Physics Essay items. However, the reliability coefficient of 0.91 and 0.90 respectively. The finding revealed that there was no central tendency effect error committed by individual raters when rating complex problem-solving skills in WAEC and NECO 2020 physics Essay Items. Also, reveals there is a significant difference at the rater's group level exhibiting central tendency effect when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items which implies that there is no group-level central tendency effect present among raters when rating complex problem-solving skills between WAEC and NECO 2020 physics Essay Items. The researchers concluded that rater effects are sources of variance in performance ratings that are associated with the rater's behaviour and not the actual performance of the ratee. It was recommended that to prevent the central tendency effect, it is important for superior raters to regularly observe their subordinates throughout the evaluation period and keep notes for future reference.

Rater Severity/Leniency and Bias in EFL Students' Composition Using Many-Facet Rasch Measurement (MFRM)

<p>The study aims to investigate the extent to which raters exhibit tendencies towards being overly severe, lenient, or even bias when evaluating students' writing compositions in Indonesia. Data were collected from 15 student essays and four raters with master's degrees in English education. The Many-facet Rasch measurement (MFRM), automatized by Minifac software, a program created for the Many-facet Rasch measurement, was used for data analysis. This was done by meticulously dissecting the assessment process into its distinct components—raters, essay items, and the specific traits or criteria being evaluated in the writing rubric. Each rater's level of severity or leniency, essentially how strict or lenient they are in assigning scores, is scrutinized. Likewise, the potential biases that raters might introduce into the grading process are carefully examined. The findings revealed that, while the raters used the rubric consistently when scoring all test takers, they varied in how lenient or severe they were. Scores of 70 were given more frequently than the other score. Based on the findings, composition raters may differ in how they rate students which potentially leading to student dissatisfaction, particularly when raters adopt severe scoring. The bias in scoring has highlighted that certain raters consistently tend to inaccurately score items, deviating from the established criteria (traits). Furthermore, the study also found that having more than four items/criteria (content, diction, structure, and mechanic) is essential to achieve a more diverse distribution of item difficulty and effectively measure students' writing abilities. These results are valuable for writing departments to improve the oversight of inter-rater reliability and rating consistency. To address this issue, implementing rater training is suggested as the most feasible method to ensure more dependable and consistent evaluations.</p>

Analysis of Peer and Self-Assessments Using the Many-facet Rasch Measurement Model and Student Opinions

The aim of this study is to analyze the peer and self-assessments of higher education students' oral presentation skills with the many-facet Rasch measurement model and to determine students' opinions on peer and self-assessment. In the study, convergent parallel method, one of the mixed-method research approaches, was used. The study group consisted of 11 university students studying at a state university in the 2022-2023 academic year. The FACETS program was used to analyze the data. The three facets identified in the study were the assessee (11 students), the assessor (11 students) and the items (16 items). Therefore, 11 participants scored (peer and self-assessment) on a 16-item assessment form. In addition, students' opinions on peer and self-assessment were obtained through three open-ended interview questions prepared by the researcher. According to the results of the study, it was determined that there was a statistically significant difference between the students in terms of their oral presentation skills, between the assessors in terms of their strictness/generosity in scoring, and between the criteria (items) in terms of the level of difficulty in realization. In addition, the participant opinions obtained from each interview question were analyzed through themes and sub-themes formed according to the general thoughts on peer and self-assessment, experiences, and whether the participants considered themselves as a reliable rater or not. In terms of practice, it can be suggested to provide detailed and enlightening information to students before peer and/or self-assessment in the classroom environment, and to give quick feedback to those who have not done the assessment appropriately. In addition, the reasons for the biases identified in peer and self-assessments in the current study can be investigated in future studies.

The development of formative assessment rubrics for enhancing students’ performance on Thai percussion instruments

This research aimed to develop formative assessment rubrics for enhancing students’ performance on Thai percussion instruments using the Many-Facet Rasch Measurement Partial Credit model (MFRM-PCM). Samples were high-school students playing four types of Thai instruments and the raters who were qualified and properly trained. The research instrument was rubrics used for assessing students’ Thai percussion instruments skills. Each item contained five quality levels. The research methodology was divided into three phases, that is, the synthesis and verification of content validity, pre-testing of the psychometric properties and the assessment design, and the verification of the rubrics’ psychometric properties. Four facets of the MFRM-PCM including students, raters, musical instruments, and items facet were employed to analyze the psychometric properties of the developed rubrics. It was found that the rubrics developed for assessing performances of Thai percussion instruments with the MFRM-PCM contained eight criteria and 12 items. The quality levels of the items after the category’s effectiveness analysis were reduced ranging from two to three quality levels.

Repeating the Listening Text: Effects on Listener Performance, Metacognitive Strategy Use, and Anxiety

AbstractIn second language listening assessment and pedagogy, practitioners hold different views on whether to repeat a listening text in contexts where inferences about listening ability are to be drawn from task performance. To address this issue, we investigated the effects of repeating the listening text (double play) on listener performance, listening strategies, test‐taking strategies, test‐taking anxiety, and listening anxiety. Three hundred six Austrian secondary school students responded to four listening tasks drawn from the Austrian Matura exam and completed questionnaires measuring strategic behavior and anxiety in a counter‐balanced research design. Data were analyzed using many‐facet Rasch measurement (MFRM), factor analysis, and inferential statistics. Findings confirmed that double play led to higher levels of listener performance across two task types (multiple‐choice items and note completion), however, scores were higher in the single play condition compared to the first play of double play. Students also reported lower levels of anxiety, and the use of more listening strategies and fewer test‐taking strategies in double play compared to single play with small effect sizes. We discuss the importance of balancing an empirically derived understanding of the effects of repeating the listening text with considerations of the purpose of an assessment, and the characteristics of the target language use domain.

The Impact of Topic Selection on Lexico-Grammatical Errors and Scores in English Oral Proficiency Interviews of Korean College Students

The primary objective of this study is to identify the types of errors made by Korean college students in an oral proficiency interview in relation to specific task topics, and to examine how these errors affect their lexico-grammatical proficiency scores. Ninety-six two-minute-long audio clips of 32 Korean college students on three different topics were transcribed. Lexico-grammatical errors were then coded for statistical analysis and lexico-grammatical scores were estimated using many-facet Rasch measurement analysis with two raters. Friedman tests and Wilcoxon signed-rank tests showed that noun phrase, verb phrase, and prepositional phrase errors were more frequently found with the descriptive tasks than compare-and-contrast or hypothetical prompts. A hierarchical multiple regression analysis showed that noun phrase and verb phrase errors accounted for 22% of the variance in lexico-grammatical scores. Adding utterance length variables to the initial regression model explained an additional 43% of the variance in the lexico-grammatical scores. These findings suggest that noun phrase errors and verb phrase errors should be a priority in English classes, and that it is beneficial to teach English speaking skills in a way that takes into account the task characteristics and contextual factors.

Studying From Home During COVID-19: A Many-Facets Rasch Model (MFRM) Analysis

This study proposes to measure students’ grammar distancing while studying from home and explore their strengths and weaknesses with a grammar test. It is a descriptive quantitative study, the participants being 55 students in their fifth (5th) semester of the 2020/2021 academic year at a university in Tanjungpinang, Riau Islands, Indonesia. Data is retrieved from the grammar test and analyzed using the Many-Facet Rasch Measurement (MFRM) Model Software Version 3.83.3. This research highlights that studying from home, or distance learning, impacts grammar distancing as seen in the students’ varying achievements in grammar, the breakdown being 10.91% lower achievement, 5.45% low achievement, 27.27% high achievement, 43.64% higher achievement and 12.73% highest achievement. The results also indicate their strength in sentence translations and jumbled words; 83.64% of the students displayed good understanding. Their weakness lies in rearranging compound-complex sentences with phrasal verbs. The students’ different achievements in grammar demonstrate the need for improvement for future use and better student performance.

Does Sparseness Matter? Examining the Use of Generalizability Theory and Many-Facet Rasch Measurement in Sparse Rating Designs.

Sparse rating designs, where each examinee's performance is scored by a small proportion of raters, are prevalent in practical performance assessments. However, relatively little research has focused on the degree to which different analytic techniques alert researchers to rater effects in such designs. We used a simulation study to compare the information provided by two popular approaches: Generalizability theory (G theory) and Many-Facet Rasch (MFR) measurement. In previous comparisons, researchers used complete data that were not simulated-thus limiting their ability to manipulate characteristics such as rater effects, and to understand the impact of incomplete data on the results. Both approaches provided information about rating quality in sparse designs, but the MFR approach highlighted rater effects related to centrality and bias more readily than G theory.

Investigating rater–student interaction, gender bias, and major bias in the assessment of research seminar presentation

The study aims to investigate the severity of raters toward examinees' performances and to analyze biases subject to genders and academic majors based on the rater–examinee interactions. Data were collected across a 13-week research seminar course in English. Six raters were selected to rate 33 examinees using 18-item criteria. The many-facet Rasch measurement was utilized to analyze rater–examinee interactions and bias factors. The results confirmed that the instrument is reliable and valid. This study depicted the interaction between raters and other facets using a variable map, where the raters exhibited different levels of severity/leniency in scoring students' performances in oral presentations at the research seminar course. The result based on the Rasch analysis also confirmed that gender and academic majors contaminated rater assessment. Bias interaction between raters and student gender was detected, and Rater 6 displayed bias based on gender due to the tendency to give higher scores to female than male participants with a target contrast of 2.05 logits. Bias interaction between rater academic majors and student academic majors was also identified among raters with linguistics and psychology majors.