Many-facet Rasch Measurement Research Articles

Examining the validity of an analytic rating scale for a Spanish test for academic purposes using the argument-based approach to validation

Abstract Rating scales are used to assess the performance of examinees presented with open-ended tasks. Drawing on an argument-based approach to validation, this study reports on the development of an analytic rating scale designed for a Spanish test for academic purposes. The study is one of the first that sets out the detailed scale development and validation activities for a rating scale for Spanish as a second language. The rating scale was grounded in a communicative competence model and developed and validated over two phases. The first version was trialed by five raters, and its quality was analyzed by means of many-facet Rasch measurement. Based on the raters’ experience and on the statistical results, the rating scale was modified and a second version was trialed by six raters. After the rating process, raters were sent an online questionnaire in order to collect their opinions and perceptions of the rating scale, the training and the feedback provided during the rating process. The results suggest the rating scale was of good quality and raters’ comments were generally positive, although they mentioned that more samples and training were needed. The study has implications for rating scale development and validation for languages other than English.

Rater variability across examinees and rating criteria in paired speaking assessment

This study investigates rater variability with regard to examinees’ levels and rating criteria in paired speaking assessment. 12 raters completed rater training and scored 102 examinees’ paired speaking performances using analytical rating criteria that reflect various features of paired speaking performance. The raters were fairly consistent in their overall ratings, but differed in their severity. The bias analyses using many-facet Rasch measurement revealed that a higher level of rater bias interaction was found for the rating criteria compared to those of the examinees’ levels and the pairing type which reflects a level difference between two examinees. In particular, the most challenging rating category Language Use attracted significant bias interactions. However, the raters did not display more frequent bias interactions based on the interaction-specific rating categories, such as Engaging with Interaction and Turn Organization. Furthermore, the raters tended to reverse their severity patterns across the rating categories. In the rater and examinee bias interactions, the raters tended to show more frequent bias toward the low-level examinees. However, no significant rater bias was found based on the pairing type that consisted of high-level and low-level examinees. These findings have implications for rater training in paired speaking assessment.

Investigating EFL listening comprehension skills: An empirical validation of C1 level test scores

Testing listening comprehension skills is a difficult task because of the complex nature of the listening process. This complexity also makes the process of listening comprehension test design challenging. Overcoming such challenges was one of the major tasks in a project at a major Hungarian university, during which four test‑based listening practice booklets, targeting the C1 proficiency level, were designed. The present study focuses on the investigation of the reliability of the items in the first practice booklet by analysing and empirically validating the test scores of 98 test‑takers. The test in the first practice booklet contained 30 items with four different item formats. Data analysis follows a quantitative approach. The reliability of the test scores was examined with the help of Iteman software using the approaches of classical test theory and Facets software using Many‑Facet Rasch Measurement (MFRM). The latter method was also used to empirically validate the test scores by identifying misfitting test‑takers and items in the dataset. The empirical validation with MFRM offered a subtle way of strengthening the reliability of the test scores by artificially connecting the dataset. Even though the results of the present study could be improved by pre-testing the remaining three practice tests, and by removing altogether six misfitting test‑takers and four misfitting items from the present dataset, a higher degree of reliability has been reached as far as the fitting test items are concerned. The results indicate that applying MFRM for the empirical validation of test-scores might be beneficial not only for validating listening comprehension test scores but also for validating other types of test‑scores, especially in large‑scale testing.

The CEFR Rating Scale Functioning: An Empirical Study on Self- and Peer Assessments

One of the criticisms on the Common European Framework of Reference (CEFR) rating scales pertains to the lack of reference to the performance of learners in the construction process of the scales. Therefore, this study attempted to delve into rating scale functioning by English as a Second Language (ESL) learners during self-assessment and peer assessment of their oral proficiency practice. Two objectives guided the study: 1) to gauge the overall rating scale functioning and 2) to measure each criterion scaling structure. Three self- and peer assessments’ cycles were conducted in three months. In each cycle, eleven learners recorded their own speech, uploaded their video clips to a private YouTube channel and assessed their own videos as well as selected peers based on five CEFR oral assessment criteria with six levels of ratings (A1-C2). Findings revealed that four of the CEFR levels were utilised (B1-C2). Categories A1 and A2 (basic user level) however, were not observed during the practice. Analysis from the Many-Facet Rasch Measurement (MFRM) indicated that utilised categories seemed to function usefully since each category observed was advancing by more than 1.4 logits. Category B2 dominated four criteria of ratings awarded while B1 dominated the rating distribution for fluency. The implications of this study will be discussed in relation to rating scale development, specifically on matching learners’ proficiency to the psychometrically developed rating criteria as well as illustrating assessment as learning approach in the ESL classroom where learners become the key assessors for their own performance.

Improving the Validity of L2 Performance Assessments: Use of Many-Facet Rasch Measurement

Improving the Validity of L2 Performance Assessments: Use of Many-Facet Rasch Measurement

The Self, the Peer and the Teacher in the EFL Pronunciation Class: A Comparative Study on Assessment, Perceptions and Systematicity

This pilot study is aimed at describing, analysing and comparing self, peer and teacher quantitative and qualitative assessment (and consequently also perceptions and degree of systematicity when assessing) in the English as a Foreign Language (EFL) pronunciation class in tertiary education. Accordingly, the main objectives of this study have been to measure, rank and compare the harshness or leniency of the three different types of raters involved and their consistency/systematicity when in this role, as well as to measure the levels of coincidence and/or discrepancy when evaluating from different roles and depending on quantitative or qualitative considerations. The method used for the analysis involves the quantitative and qualitative analyses of the scores obtained in a triple-role assessment task carried out by 16 students and a teacher. The calculation of various statistical measures, the triangulation of the data obtained and the Many-Facet Rasch Measurement analysis of the results have completed the study and constitute the departure point for further larger studies. From the results obtained, it can be highlighted that quantitatively, the self and the teacher’s perceptions seem to be the ones that are more distant or different, whereas the self and the peer's tend to be the most similar. In the same way, qualitative assessments seem to be more lenient, that is to say, slightly higher in mean score and have a lower coefficient of variation than quantitative ones in the three groups analysed. Consistency/systematicity is relatively high but it is still an aspect to be improved on the part of most raters.

Understanding the Role of Likeability in the Peer Assessments of University Students’ Oral Presentation Skills: A Latent Variable Approach

ABSTRACTThe purpose of the present study was twofold: (a) it examined the relationship between peer-rated likeability and peer-rated oral presentation skills of 96 student presenters enrolled in a science communication course, and (b) it investigated the relationship between student raters’ severity in rating presenters’ likeability and their severity in evaluating presenters’ skills. Students delivered an academic presentation and then changed roles to rate their peers’ performance and likeability, using an 18-item oral presentation scale and a 10-item likeability questionnaire, respectively. Many-facet Rasch measurement was used to validate the data, and structural equation modeling (SEM) was used to examine the research questions. At an aggregate level, likeability explained 19.5% of the variance of the oral presentation ratings and 8.4% of rater severity. At an item-level, multiple cause-effect relationships were detected, with the likeability items explaining 6–30% of the variance in the oral presentation items. Implications of the study are discussed.

Investigating Central Tendency in Competency Assessment of Design Electronic Circuit: Analysis Using Many Facet Rasch Measurement (MFRM)

Investigating Central Tendency in Competency Assessment of Design Electronic Circuit: Analysis Using Many Facet Rasch Measurement (MFRM)

Measurement of Early Science and Mathematics Standard Instrument: Performance Assessment and psychometric setting using ZPD Concept

Objective: The purpose of this study is to measure an instrument that could assess Early Science and Mathematics Standard in the domain of science and technology. Children’s measurement standards in early years include the performance in thinking skills. The design is a combination of the theoretical framework representing Early Sciences and Mathematics Standard and Zone of Proximal Development (ZPD). Method/Analysis: We developed a performance assessment standard and scoring set (rubric) for measuring children’s responses. Participants in this study were 30 children of 2-year old and 30 children who were 3 years old. These children were from several child-care centres. 21 items on scientific attitude, scientific skills, investigate the nature of life, pre-number experience, concepts of number, shapes and space, and construction were assessed. Mathematical models such as the Rasch model have provided useful representations of social science problems where it coordinates data with the requirements of a useful definition of measurement. Findings: The study shows the Many-facet Rasch Measurement (MFRM) an extended version of the Rasch Model techniques which is combined with children’s measurement standards to examine the validity and reliability of scores for the performance rating scale. Additionally, an infit statistic (.76) for 2 years and (1.46) for 3 years for higher responses support the validity of scores. For reliability, the person reliability is good at .99, rater reliability is at .97, domain reliability is good at 1.00 and the item reliability is at .96 for overall scoring measure. Application/Improvement: This paper provides educators and researchers with a useful tool to facilitate measurement in early childhood years. It gives a great recommendation where the instrument is valid and reliable.

Calibrating Communication Competencies

The Many-faceted Rasch measurement model is used in the creation of a diagnostic instrument by which communication competencies can be calibrated, the severity of observers/raters can be determined, the ability of speakers measured, and comparisons made between various groups.

The Multiple Mini-Interview as an Admission Tool for a PharmD Program Satellite Campus

Objective. To assess the multiple mini-interview (MMI) as an admission tool for a satellite campus. Methods. In 2013, the MMI was implemented as part of a new admissions model at the UNC Eshelman School of Pharmacy. From fall 2013 to spring 2015, 73 candidates were interviewed by 15 raters on the satellite campus in Asheville, North Carolina. A many-facet Rasch measurement (MFRM) with three facets was used to determine the variance in candidate ratings attributable to rater severity, candidate ability, and station difficulty. Candidates were surveyed to explore their perceptions of the MMI. Results. Rasch measures accounted for 48.3% of total variance in candidate scores. Rater severity accounted for 9.1% of the variance, and candidate ability accounted for 36.2% of the variance. Eighty percent of survey respondents (strongly) agreed that interviewers got to know them based on questions they answered.Conclusion. This study suggests that the MMI is a useful and valid tool for candidate selection at a satellite campus.

Development and initial argument-based validation of a scoring rubric used in the assessment of L2 writing electronic portfolios

Abstract Although writing electronic portfolios (ePortfolios) help learners communicate digitally and provide a platform for works to be better collected and presented, challenges are present in their assessment. This paper reports the development and validation of a writing ePortfolio scoring rubric for an action research course for L2 learners. Using Bachman’s (2005) ‘Assessment Use Argument’ as a basis, two main claims to support the validity of the rubric were examined: (1) the rubric is a reliable tool; and, (2) the rubric is relevant to the construct being measured. A mixed-method approach was used in the development and validation of the eight-criteria analytic and holistic evaluation scoring rubric. Thirteen raters evaluated thirty-eight ePortfolios in the study. The analyses of raters’ ratings using a many-facet Rasch measurement approach and raters’ individual standardised open-ended interviews indicated that overall the rubric’s analytic categories functioned appropriately to assess the intended construct. However, the criterion ‘Ease of Navigation’ was found to be misfitting due to differences in raters’ evaluation of the same ePortfolios. Overall, the argument-based validation indicated that the scoring rubric is a reliable and valid instrument for the purpose of assessing L2 writing ePortfolios in the context for which it was developed.

“I can see that”: Developing shared rubric category interpretations through score negotiation

Abstract Performance assessments using raters will always contain some subjectivity, and disagreement among raters necessitates reliable methods for resolving scores. Negotiation is one effective method to guide scoring decisions and reduce raters’ tendencies to be unexpectedly severe or lenient when scoring specific rubric categories or examinees. Beyond its utility for scoring, however, negotiation is also a resource for raters to co-construct interpretations about the language constructs being measured. This study uses quantitative and qualitative methods to trace how negotiation impacts raters’ scoring decisions and examine in detail how raters develop joint interpretations of rubric category criteria. Scores from the writing section of a high stakes English language placement exam (n = 60) were analyzed using ANOVA and many-faceted Rasch measurement to determine which categories were frequently assigned discrepant scores and to estimate rater severity. Discourse analysis of six audiotaped negotiation sessions was then used to examine how raters’ understanding of rubric criteria converged over time. Our results indicate that through negotiation, raters used shared terminology and justifications to clarify ambiguous constructs and work to establish shared values. The results suggest that score negotiation influences scoring inferences and also creates affordances for raters to ground those inferences in shared constructions of meaning.

A Many-Facet Rasch analysis comparing essay rater behavior on an academic English reading/writing test used for two purposes

Abstract Second language (L2) writing researchers have noted that various rater and scoring variables may affect ratings assigned by human raters (Cumming, 1990; Vaughan, 1991; Weigle, 1994, 1998, 2002; Cumming, Kantor, & Powers, 2001; Lumley, 2002; Barkaoui, 2010). Contrast effects (Daly & Dickson-Markman, 1982; Hales & Tokar, 1975; Hughes, Keeling, & Tuck, 1983), or how previous scores impact later ratings, may also color raters’ judgments of writing quality. However, little is known about how raters use the same rubric for different examinee groups. The present paper concerns an integrated reading and writing test of academic English used at a U.S. university for both admissions and placement purposes. Raters are trained to interpret the analytic scoring rubric similarly no matter which test type is scored. Using Many-Facet Rasch measurement (Linacre, 1989/1994), I analyzed scores over seven semesters, examining rater behavior on two test types (admissions or placement). Results indicated that, of 25 raters, five raters showed six instances of statistically significant bias on admissions or placement tests. The findings suggest that raters may be attributing scores to a wider range of writing ability levels on admissions than on placement tests. Implications for assessment, rater perceptions, and small-scale academic testing programs are discussed.

Determining cloze item difficulty from item and passage characteristics across different learner backgrounds

Cloze tests have been the subject of numerous studies regarding their function and use in both first language and second language contexts (e.g., Jonz & Oller, 1994; Watanabe & Koyama, 2008). From a validity standpoint, one area of investigation has been the extent to which cloze tests measure reading ability beyond the sentence level. Using test data from 50 30-item cloze passages administered to 2,298 Japanese and 5,170 Russian EFL students, this study examined the degree to which linguistic features for cloze passages and items influenced item difficulty. Using a common set of 10 anchor items, all 50 tests were modeled in terms of person ability and item difficulty onto a single scale using many-faceted Rasch measurement ( k = 1314). Principle components analysis was then used to categorize 25 linguistic item- and passage-level variables for the 50 cloze tests and their respective items, from which three components for each passage- and item-level variables were identified. These six factors along with item difficulty were then entered into both a hierarchical structural equation model and a linear multiple regression to determine the degree to which difficulty in cloze tests could be explained separately by passage and item features. Comparisons were further made by looking at differences in models by nationality and by proficiency level (e.g., high and low). The analyses revealed noteworthy differences in mean item difficulties and in the variance structures between passage- and item-level features, as well as between different examinee proficiency groups.

Measuring the impact of rater negotiation in writing performance assessment

Previous research in second language writing has shown that when scoring performance assessments even trained raters can exhibit significant differences in severity. When raters disagree, using discussion to try to reach a consensus is one popular form of score resolution, particularly in contexts with limited resources, as it does not require adjudication by at third rater. However, from an assessment validation standpoint, questions remain about the impact of negotiation on the scoring inference of a validation argument (Kane, 2006, 2012). Thus, this mixed-methods study evaluates the impact of score negotiation on scoring consistency in second language writing assessment, as well as negotiation’s potential contributions to raters’ understanding of test constructs and the local curriculum. Many-faceted Rasch measurement (MFRM) was used to analyze scores ( n = 524) from the writing section an EAP placement exam and to quantify how negotiation affected rater severity, self-consistency, and bias toward individual categories and test takers. Semi-structured interviews with raters ( n = 3) documented their perspectives about how negotiation affects scoring and teaching. In this study, negotiation did not change rater severity, though it greatly reduced measures of rater bias. Furthermore, rater comments indicated that negotiation supports a nuanced understanding of the rubric categories and increases positive washback on teaching practices.

Topic and background knowledge effects on performance in speaking assessment

This study explores the extent to which topic and background knowledge of topic affect spoken performance in a high-stakes speaking test. It is argued that evidence of a substantial influence may introduce construct-irrelevant variance and undermine test fairness. Data were collected from 81 non-native speakers of English who performed on 10 topics across three task types. Background knowledge and general language proficiency were measured using self-report questionnaires and C-tests respectively. Score data were analysed using many-facet Rasch measurement and multiple regression. Findings showed that for two of the three task types, the topics used in the study generally exhibited difficulty measures which were statistically distinct. However, the size of the differences in topic difficulties was too small to have a large practical effect on scores. Participants’ different levels of background knowledge were shown to have a systematic effect on performance. However, these statistically significant differences also failed to translate into practical significance. Findings hold implications for speaking performance assessment.

수학 창의력 문제 해결 검사에서의 일반화가능도 이론과 다국면 라쉬모형의 비교 연구

This study describes the use of Generalizability Theory (GT) and Many-Facet Rasch Measurement (MFRM) to evaluate and improve the rating procedure in a mathematics creative problem solving test. Results indicate that these two methods agree about the relative degrees of variation among the facets but slightly differ on how to account for the sources of variation. For both the GT and MFRM results, the variance component for the Person by Item interaction is relatively large, indicating significant variability. Results from both methods also indicated that variance due to rater and interactions related with rater were relatively low. The reliability of the mean rating for each examinee based on five items, four raters and four rating criteria using a fully crossed design was 0.58(G-coefficient) and 0.49(phi coefficient). We found the guidelines from the Decision study (D-study) to obtain a more optimal reliability coefficients, it needed at least ten items. Depending on the purpose of a particular study, GT or MFRM may be the appropriate measurement technique to use.

The Cross-Cultural Invariance of Creative Cognition: A Case Study of Creative Writing in U.S. and Russian College Students.

Unlike intelligence, creativity has rarely been investigated from the standpoint of cross-cultural invariance of the structure of the instruments used to measure it. In the study reported in this article, we investigated the cross-cultural invariance of expert ratings of creative stories written by undergraduate students from the Russian Federation and the United States. Analyses of differential rater and item functioning using Many-Facet Rasch Measurement and multiple levels of invariance using confirmatory factor analyses suggested partial measurement invariance of creative ability estimates obtained using this method in two cultures. Russian and U.S. students demonstrated similar overall levels of creativity; however, U.S. students received higher emotionality ratings than Russian students did. The findings are discussed in the context of viewing creativity as at least a partially culturally invariant trait whose manifestation is moderated by culture-specific semantic knowledge and patterns of linguistic behavior.

Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn

This study investigated the application of WriteToLearn on Chinese undergraduate English majors’ essays in terms of its scoring ability and the accuracy of its error feedback. Participants were 163 second-year English majors from a university located in Sichuan province who wrote 326 essays from two writing prompts. Each paper was marked by four human raters as well as WriteToLearn. Many-facet Rasch measurement (MFRM) was conducted to calibrate WriteToLearn’s rating performance in scoring the whole set of essays against those of four trained human raters. The accuracy of WriteToLearn’s feedback on 60 randomly selected essays was compared with the feedback provided by human raters. The two main findings related to scoring were that WriteToLearn was more consistent but highly stringent relative to the four trained human raters in scoring essays and that it failed to score 7 essays. In terms of error feedback, WriteToLearn had an overall precision and recall of 49% and 18.7% respectively. These figures did not meet the minimum threshold of 90% precision for it to be a reliable error detecting tool set by Burstein, Chodorow, and Leacock (2003). Furthermore, it had difficulty in identifying the errors made by Chinese undergraduate English majors in the use of articles, prepositions, word choice and expression.