A Comparison of the Results of Many-Facet Rasch Analyses Based on Crossed and Judge Pair Designs

Abstract

The many-facet Rasch model (MFRM) is an extension of the item response theory, a one-parameter model. The one-parameter model, also known as the simple Rasch model, was created by George Rasch (1960). It includes two facets: ability level and item difficulty (Stahl, Bergstrom, Shumway, & Fisher, 1997). The probabilistic relationship between the two facets is formulated using the following equation:log(Pni1 / Pni0) = Bn - DiIn this equation, Bn shows the ability level of the examinee with the number n, Di shows the difficulty level of the item with the number i, Pni1 shows the possibility of the examinee number n scoring 1 point on item number i, and Pni0 shows the possibility of the examinee number n scoring 0 points on item number i (Schumacker, 1996). This mathematical model makes it possible to estimate the ability level and item difficulty simultaneously (Brown, O'Gorman, & Du, 1996). As the equation shows, the simple Rasch model was created for dichotomous items that can be rated as either correct or incorrect (Sebok, Luu, & Klinger, 2013). Andrich (1978) extended the use of the simple Rasch model to Likert-type data, which produced the rating scale model, and Masters (1982) created the partial credit model, which was an extension of the research by Andrich (1978). The partial credit model was developed to analyze open-ended items. In this model, researchers can conduct a partial rating considering the steps followed in the solution process instead of scoring the items as either correct or incorrect (Cagnone & Ricci, 2005). However, both the rating scale model by Andrich (1978) and the partial credit model by Masters (1982) consist of two facets-ability level and item difficulty-like the simple Rasch model. Following the studies conducted by Masters (1982), Linacre (1989) furthered the partial credit model by including variability sources such as judge effect in the model (Farrokhi & Esfandiari, 2011). By adding variability sources, which could affect the assessment results, to the partial credit model, he developed a new model called the many-facet Rasch model (MFRM) (Mulqueen, Baker, & Dismukes, 2000). Table 1 presents a summary of the different types of Rasch models along with their characteristics and the researchers who created them.The Many-Facet Rasch Model (MFRM)In general, the MFRM includes three variability sources: examinees, items, and judges. These sources are shown in the equations below. In this equation explaining the MFRM, Bn shows the ability level of examinee n, Di shows the difficulty level of item i, Cj shows the severity or leniency of the judge j, and Fk shows the difficulty of observing category k relative to category (k-1) (note that F is not a separate facet but a part of the item facet). The expression on the left side of the equation shows the possibility for examinee n whose response to item number i was rated by judge j to obtain a point corresponding to the category k and the possibility of this examinee to obtain a score corresponding to the category (k-1) (Linacre, 1991).log(Pnijk / Pnij(k-1)) = Bn - Di - Cj - FkThough the general form of the MFRM consists of three facets (examinee, item, and judge), this does not mean the number of facets in the model cannot be increased. In other words, if there are any other sources of variability (other than examinee, item and judge) that can affect the assessment results, these sources with the potential to affect the assessment results can be added to the model. For instance, the scale used for rating is also included in the model as a source of variability that can affect the assessment results if there are different rating scales used for different items of the test (Hung, Chen, & Chen, 2012). This is the equation for a model of the rating scale affecting the assessment results:log(Pnijk / P(nij(k-1)) = Bn - Di- Cj - FikIn this equation, Fik represents the difficulty of observing category (k-1) relative to category k for item number i. …