Many-faceted Rasch Analysis Research Articles

Automated Sign Language Vocabulary Assessment: Comparing Human and Machine Ratings and Studying Learner Perceptions

ABSTRACT Although automated spoken language assessment is rapidly growing, such systems have not been widely developed for signed languages. This study provides validity evidence for an automated web application that was developed to assess and give feedback on handshape and hand movement of L2 learners’ Swiss German Sign Language signs. The study shows good machine-internal and human-machine agreement through many-facet Rasch analysis. Learner perceptions examined through questionnaire responses indicate that the automated system occasionally generated ratings which impacted the quality of feedback at the level of individual signs for individual learners. Implications are discussed from a learning-oriented assessment perspective.

A validation study of a consecutive interpreting test using many-facet Rasch analysis

Validation is the process of conducting a validity study on a test, gathering evidence to support inferences made based on test scores. Using many-facet Rasch analysis this paper validates the result of a consecutive interpreting test from four aspects: rating scale, raters, examinees and interpreting tasks. The study also identifies factors that affect the test results through bias analysis. The results of this analysis show that: (1) this analytical rating scale and task design can effectively reflect the examinees' interpreting competence, thus making it suitable for validating interpreting assessments; (2) one of the sub-scales: interpreting skills and manners, has a slightly negative effect on the overall validity of the assessment, making further revision on this sub-scale desirable.

Developing and evaluating a contextualized interactional competence rating scale based on a metaphorical conceptualization

Abstract Rating scales developed to measure interactional competence (IC) are mainly data-driven which can incur unaffordable costs for assessment practitioners with limited resources, such as universities using this test for placement purposes. A cheaper alternative is to use models proposed in the literature to develop such rating scales. This mixed-methods study evaluates a contextualized analytic IC rating scale developed based on Galaczi and Taylor’s (2018) metaphorical conceptualization of IC. A many-facet Rasch analysis indicated this rating scale could separate examinees into three levels of ability, which was lower than the expected four levels. To explore this finding, raters were interviewed about their experiences of using this rating scale. The findings show the potential and limitations of developing contextualized IC ratings scales based on literature and possible avenues for future research. They also suggest a direct interaction between assessment and instructional practices of IC with implications for language teaching.

Content Analysis of Test Taker Responses on an AP Japanese Computer-Simulated Conversation Test: A Mixed Methods Approach for A Validity Argument

ABSTRACT The present study is part of a larger mixed methods project that investigated the speaking section of the Advanced Placement (AP) Japanese Language and Culture Exam. It investigated assumptions for the evaluation inference through a content analysis of test taker responses. Results of the content analysis were integrated with those of a many-facet Rasch analysis of the same speech data. This study found that most information-seeking prompts elicited a good sized ratable speech sample with relevant content, and the rating criteria seemed to fit with the nature of the interaction. Therefore, information-seeking prompts generally provided appropriate evidence of test takers’ ability. In contrast, non-information-seeking prompts such as requests and expressive prompts tended to have issues with eliciting a good sized ratable speech sample with relevant content, and their response expectations realized in the rating criteria did not fit with the nature of the interaction. Thus, non-information-seeking prompts showed greater potential of becoming sources of measurement error with the current test design. This article discusses possible solutions to increase the validity of the evaluation inference. Findings from the present study would be useful for future test development of computer-based L2 tests that aim to assess interpersonal communication skills.

Can L2 course duration compensate for the impact of demographic and educational background variables on second language writing development?

This article focuses on equity as inclusion in an adult L2 education system. Using a real-world high-stakes writing test, we examine to what extent learners with different educational backgrounds partaking in the same educational system have the same opportunity of achieving the same minimal standard. In the context of the current research, that minimal proficiency standard is the A2 level of the Common European Framework of Reference for Languages (CEFR). Achieving that level is a requirement for all non-EU migrants to Flanders (Belgium), where the study was conducted. The research population consisted of 1056 adults attending three different L2 learning tracks of Dutch as a second language in Belgium. Each track caters to learners with different educational backgrounds, ranging from primary to tertiary education. Using logistic multilevel regression and many-facet Rasch analysis, this study compared writing performances of the different learning tracks. The results show significant differences in test scores and pass probabilities at the end of the different A2 tracks, as well as very different learner profiles in terms of syntactic and lexical complexity, and in terms of error types.

Many-faceted rasch calibration of the foot function index-revised short form

BackgroundThe Foot Function Index Revised Short version (FFI-RS) is a foot- and ankle- patient-reported outcome measure (PROM), developed from the Foot Function Index (FFI). Previous studies, estimating item parameters and multidimensional properties, have limitations properly establishing the measurement properties of the FFI-RS. A multi-faceted Rasch analysis with a larger sample would allow for a more robust validation approach to improve the clinical interpretation of the FFI-RS using a multidimensional perspective. Therefore, the purpose of this study was to assess the psychometric properties of the FFI-RS as a PROM of foot function.MethodA total of 2184 patients with foot pathology who completed the FFI-RS were included in the data. Data were extracted from the cloud-based orthopedic and sports medicine global registry Surgical Outcome System (SOS). The psychometric properties of the FFI-RS were assessed using a many-faceted Rasch analysis that included model-data fit, rating scale function, item-person map (distribution of item difficulty and person ability), and item difficulty of the subscale.ResultsTwo misfit items were discovered and deleted; 32-items from the original FFI-RS were retained. The 4-item Likert scale functioned effectively and item difficulty (-0.58 to 1.48), subscale difficulty (-0.58 to 1.15), and person’s foot function (-6.62 to 6.24) had wide distributions.ConclusionsMany-faceted Rasch analysis revealed the FFI-RS had sound psychometric properties using the many-faceted Ranch analysis and retained 32 of the original items. Clinicians and researchers should consider weaknesses identified with items in the ‘Difficulty” subscale and future work should be conducted to modify or develop items that will more accurately evaluate a wide range of foot function levels.

Gauging the impact of literacy and educational background on receptive vocabulary test scores

The Peabody Picture Vocabulary Test (PPVT) is a widely used test of receptive vocabulary, but no researchers to date have examined the performance of low-educated, low-literate L2 adults, or compared these individuals’ performances to their more highly educated peers. In this study, we used many-facet Rasch analysis and mixed-effects linear regression to determine the impact of educational background and other demographic variables on PPVT test performance. The analyses rely on the performance data of 1,014 adult learners of Dutch as a second language on the Dutch version of the PPVT (PPVT-III-NL). The results show that a substantial proportion of score variance can be attributed to educational background variables and to the educational tracks the participants followed. These tracks, which cater to the needs of different L2 learner profiles, appear to exacerbate rather than mediate any performance differences. Although this study provides evidence of performance differences and differential item functioning resulting from linguistic, demographic, and educational variables, it offers no data to invalidate the use of the PPVT on low-educated L2 adults.

Examining severity and centrality effects in TestDaF writing and speaking assessments: An extended Bayesian many-facet Rasch analysis

Severity and centrality are two main kinds of rater effects posing threats to the validity and fairness of performance assessments. Adopting Jin and Wang’s (2018) extended facets modeling approach, we separately estimated the magnitude of rater severity and centrality effects in the web-based TestDaF (Test of German as a Foreign Language) writing and speaking assessments using Bayesian MCMC methods. The findings revealed that (a) the extended facets model had a better data–model fit than models that ignored either or both kinds of rater effects, (b) rating scale and partial credit versions of the extended model differed in terms of data–model fit for writing and speaking, (c) rater severity and centrality estimates were not significantly correlated with each other, and (d) centrality effects had a demonstrable impact on examinee rank orderings. The discussion focuses on implications for the analysis and evaluation of rating quality in performance assessments.

Development and validation of the Physician's Global Assessment of Fingernail Psoriasis

Several clinician-rated scoring systems are available to assess nail psoriasis severity, but only one has been partially validated. To develop and validate the Physician's Global Assessment of Fingernail Psoriasis (PGA-F), a new clinician-rated severity scale. A literature review, concept elicitation, pilot cognitive debriefing and clinical expert consultations informed the development of the PGA-F. A multistage mixed-methods analysis consisted of practising dermatologist cognitive interviews (n=10) for instrument clarity, relevance and comprehensiveness. Inter-rater reliability (IRR) of ratings from dermatologists (n=22) and clinical trial investigators (n=8) was tested using many-facet Rasch analysis. Concurrent validity between the PGA-F and modified Nail Psoriasis Severity Index (mNAPSI) at screening and baseline was assessed along with the degree of discrimination. Intraclass correlation coefficient (ICC) for single raters at multiple assessments determined IRR. The PGA-F synthesizes severity ratings across multiple disease features that classify individuals into 1 of 5 levels (clear to severe). Cognitive interviews confirmed content validity: all (n=10, 100%) participants who agreed clinical criteria were consistent with nail psoriasis; no mismatched severity levels; and training photographs were realistic representations. All PGA-F items were locally independent and targeted patients along the severity continuum with complementary precision (item fit statistics: < the 1.5 acceptability threshold; exact agreements among the dermatologists [44%] and trial investigators [61.5%] exceeded 40% of acceptability threshold). Clinician reliability exceeded the threshold of acceptability for dermatologists and clinical trial investigators: 0.85 and 0.73, respectively. There was adequate correlation (>0.30) between mNAPSI and PGA-F at baseline and Week 26 with significant discrimination of severity and monotonic increases in the mNAPSI for each level of categorical severity on the PGA-F. ICC results for each type of IRR indicate that clinicians were consistent in individual patient ratings. The PGA-F is a rapid, valid and reliable clinician-rated severity scale for use in clinical practice and research.

Validating a partial-credit scoring approach for multiple-choice science items: an application of fundamental ideas in science

ABSTRACT This study provides a partial-credit scoring (PCS) approach to awarding students’ performance on multiple-choice items in science education. The approach is built on fundamental ideas, the critical pieces of students’ understanding and knowledge to solve science problems. We link each option of the items to several specific fundamental ideas to capture their mastery patterns when an option is selected. Using these mastery patterns to order the options of items and accordingly assign credits to students, we measure students’ cognitive proficiency without including additional measures (e.g. times of trial) to the test or requiring extra support (e.g. technology). Using many-facet Rasch analysis, we find that the ordered options students selected were aligned with their ability measures. The PCS yields robust psychometric quality; compared to the dichotomous scoring of multiple-choice items, it generates better item fit and separation parameters. Besides, this PCS approach helps address construct validity by modelling student responses at the option level to reflect students’ mastery of fundamental ideas.

Comparing student science performance between hands-on and traditional item types: A many-facet Rasch analysis

This study aimed to compare student science performance between hands-on and traditional item types by investigating the item type effect and the interaction effect between item type and science content domain. In Shanghai, China, 2404 ninth-graders from six urban junior high schools took part in the study. The partial credit many-facet Rasch measurement analysis was used to examine the instrument's quality and investigate the item type effect and the interaction effect. The results showed that the traditional item type was significantly more difficult for participants than the hands-on item type, exhibiting a moderate-to-large effect size. Moderate or large interaction effects of an item type with a specific content domain on student science performance were also detected. Students performed better on some science content domains with a particular item type (either hands-on or traditional). Implications for assessment developers and science instructors were also discussed.

Development and Construct Validation of a Diagnostic Pronunciation Rating Scale by Many-Facet Rasch Analysis

Development and Construct Validation of a Diagnostic Pronunciation Rating Scale by Many-Facet Rasch Analysis

A Rasch-Based Validation of ELT Certificate-LORT

The present study was executed with the purpose of validating ELT Certificate Lesson Observation and Report Task (ELTC-LORT), which was developed by China Language Assessment to certify China’s EFL teachers by performance-based testing. The ELT Certificate has high-stakes considering its impacts on candidates’ recruitment, ELT in China and quality of education, so it is crucially important for its validation so as to guarantee fairness and justice. The validity of task construct and rating rubric went through a process suited for many-facet Rasch measurement supplemented with qualitative interviews. Participants (N = 40) were provided with a video excerpt from a real EFL lesson, and required to deliver a report on the teacher’s performance. Two raters graded the records of the candidates’ reports using rating scales developed to measure EFL teacher candidates’ oral English proficiency and ability to analyze and evaluate teaching. Many-facet Rasch analysis demonstrated a successful estimation, with a noticeable spread among the participants and their traits, proving the task functioned well in measuring candidates’ performance and reflecting the difference of their ability. The raters were found to have good internal self-consistency, but not the same leniency. The rating scales worked well, with the average measures advancing largely in line with Rasch expectations. Semi-structured interviews as well as focus group interviews were executed to provide knowledge regarding the raters’ performance levels and the functionalities of the rating scale items. The findings provide implications for further research and practice of the Certificate.

Development and Construct Validation of a Diagnostic Pronunciation Rating Scale by Many-Facet Rasch Analysis

Development and Construct Validation of a Diagnostic Pronunciation Rating Scale by Many-Facet Rasch Analysis

Comparison of Automatic and Expert Teachers’ Rating of Computerized English Listening-Speaking Test

Through Many-Facet Rasch analysis, this study explores the rating differences between 1 computer automatic rater and 5 expert teacher raters on scoring 119 students in a computerized English listening-speaking test. Results indicate that both automatic and the teacher raters demonstrate good inter-rater reliability, though the automatic rater indicates less intra-rater reliability than college teacher and high school teacher raters under the stringent infit limits. There&amp;rsquo;s no central tendency and random effect for both automatic and human raters. This research provides evidence for the automatic rating reform of the computerized English listening-speaking test (CELST) in Guangdong NMET and encourages the application of MFRM in actual score monitoring.

An Empirical Study for the Statistical Adjustment of Rater Bias

This study investigated the effectiveness of statistical adjustments applied to rater bias in many-facet Rasch analysis. Some changes were first made in the dataset that did not include rater × examinee bias to cause to have rater × examinee bias. Later, bias adjustment was applied to rater bias included in the data file, and the effectiveness of the statistical adjustment was further examined. The outcomes pertaining to the datasets with and without bias, and to which the bias adjustment was applied, were compared. It was concluded that diversities created by rater × examinee bias in examinees’ ability estimation, item difficulty indices and measures of rater severity and leniency were, to a large extent, eliminated by bias adjustment. This result indicates that the bias adjustment using many-facet Rasch analysis is a viable way to control rater bias.

Evaluating Teacher Effectiveness Using Classroom Observations: A Rasch Analysis of the Rater Effects of Principals

ABSTRACTIn most U.S. schools, teachers are evaluated using observation of teaching practice (OTP). This study investigates rater effects on OTP ratings among 421 principals in an authentic teacher evaluation system. Many-facet Rasch analysis (MFR) using a block of shared ratings revealed that principals generally (a) differentiated between more and less effective teachers, (b) rated their teachers with leniency (i.e., overused higher rating categories), and (c) differentiated between teaching practices (e.g., Cognitive Engagement vs. Classroom Management) with minimal halo effect. Individual principals varied significantly in degree of leniency, and approximately 12% of principals exhibited severe rater bias. Implications for use of OTP ratings for evaluating teachers’ effectiveness are discussed. Strengths and limitations of MFR to analyze rater effects in OTP are also discussed.

Impact of adjusting for inter-rater variability in conference abstract ranking and selection processes.

Scientific conferences provide a forum for clinicians, educators, students and researchers to share research findings. To be selected to present at a scientific conference, authors must submit a short abstract which is then rated on its scientific quality and professional merit and is accepted or rejected based on these ratings. Previous research has indicated that inter-rater variability can have a substantial impact on abstract selection decisions. For their 2015 conference, the Occupational Therapy Australia National Conference introduced a system to identify and adjust for inter-rater variability in the abstract ranking and selection process. Ratings for 1340 abstracts submitted for the 2015 and 2017 conferences were analysed using many-faceted Rasch analysis to identify and adjust for inter-rater variability. Analyses of the construct validity of the abstract rating instrument and rater consistency were completed. To quantify the influence of inter-rater variability of abstract selection decisions, comparisons were made between decisions made using Rasch-calibrated measure scores and decisions that would have been made based purely on raw average scores derived from the abstract ratings. Construct validity and measurement properties of the abstract rating tool were good to excellent (item fit MnSq scores ranged from 0.8 to 1.2; item reliability index=1.0). Most raters (24 of 27, 89%) were consistent in their use of the rating instrument. When comparing abstract allocations under the two conditions, 25% of abstracts (n=341) would have been allocated differently if inter-rater variability was not accounted for. This study demonstrates that, even with a strong abstract rating instrument and a small rater pool, inter-rater variability still exerts a substantial influence on abstract selection decisions. It is recommended that all occupational therapy conferences internationally, and scientific conferences more generally, adopt systems to identify and adjust for the impact of inter-rater variability in abstract selection processes.

Assessing C2 writing ability on the Certificate of English Language Proficiency: Rater and examinee age effects

Abstract Differentiating between advanced L2 writers at the higher levels of the Common European Framework of Reference (CEFR) presents a challenge in assessment. The distinction between descriptors at the C1 and C2 levels are fine-grained, and even native speakers of a language may not consistently achieve them. At the same time, the CEFR has generally been conceived with the language abilities and contexts of use of adults in mind, thus making CEFR-based interpretations of young language learner’s abilities problematic. This study examines two issues in the assessment of C2-level writing in the context of the Certificate of English Language Proficiency (CELP) writing task: rater effects and examinee age. Interrater reliability and many-facet Rasch analysis showed that raters varied substantially in severity. CELP scoring procedures for rater disagreement partially mitigated severity differences. Contrary to expectations, age differentiated examinee abilities minimally and defied hypothesized ordering (i.e., that writing ability would increase with age). Additionally, some raters were found to demonstrate bias towards the youngest examinees. Specific implications for the CELP’s validity argument and broader implications for assessing young writers in CEFR terms are discussed.

THE EFFECTS OF TEST CHARACTERISTICS ON THE HIERARCHICAL ORDER OF READING SKILLS

Purpose – The study sought to determine the hierarchical nature of reading skills. Whether reading is a ‘unitary’ or ‘multi-divisible’ skill is still a contentious issue. So is the hierarchical order of reading skills. Determining the hierarchy of reading skills is challenging as item difficulty is greatly influenced by factors related to test characteristics. To examine the interaction between these factors and item difficulty, and determine the possibility of such a hierarchy, this study used the multifaceted Rasch approach. Methodology – In this descriptive study, a 42-MCQ reading test was administered to 944 ESL lower secondary students, randomly selected from eleven Malaysian national-type schools in the Federal Territory of Kuala Lumpur and the state of Selangor. These student populations were selected as the development of reading ability was considered critical at this stage of schooling. The reading test items were identified according to the following aspects: Reading Skill Areas (Interpreting information, making Inference, Understanding figurative language, Drawing conclusions, Scanning for details and Finding word meanings), Context Type (Linear and Non-Linear), and Text Type (Ads, Notice, Chart, Story extract, Short message, Poem, Short news report, Brochure, Formal letter, Conversation, Long passage and weather forecast). Applying the Many-Facet Rasch model of measurement, the study analyzed student responses to the test items with the help of FACETS, version 3.7.1.4. Findings – The findings showed that context types, skill areas, and text types differed in difficulty (p<.01), with those items that required understanding and interpretation being more demanding. Test items based on linear contexts were more difficult than those based on non-linear contexts. Understanding figurative language was found to be the most difficult skill followed by Making inference and Interpreting information. The easiest reading skill was Scanning for details, followed by Finding word meanings. The reading skill, Drawing conclusions, was close to the average difficulty level. The findings also indicated that texts that were longer and had more information tended to be more difficult. Significance – This study has also shed new light on the theory and practice of reading. The findings support the hierarchical nature of reading skills. Different reading skills were found to exert differential cognitive demands, and those which required higher cognitive ability were more difficult for learners to acquire and perform. Understanding the hierarchy of reading skills will help language teachers to target their teaching more effectively; course designers to produce more appropriate teaching and learning materials; and test writers to develop test items that better meet students’ reading competencies. Keywords: Reading hierarchy; reading skills; many-facet Rasch analysis; FACETS.