Many-facet Rasch Model Research Articles

A method for the definition of a self-awareness behavior dimension with clinical subjects: A latent trait analysis

The main scope of the present study was to devise a method in order to define a dimension characteristic of self-awareness behaviors with clinical subjects. To do so, I adopted a latent trait methodological approach. I studied the way patients expressed their treatment requests through their behaviors, both during their admission to a medical center in Northern Italy and after a period of treatment that involved an integrated (psychoanalytical and pharmacological) approach. The subjects were 48 females suffering from affective disorders, often combined with personality disorders. Five self-awareness indicators were identified, based both on interviews conducted with the patients and on the literature on the subject. The data gathered were analyzed by means of the many-facet Rasch model (Linacre, 1989). The results confirmed the existence of a self-awareness dimension characterized by the five indicators. Moreover, there was evidence that an improvement in self-awareness occurred during the pretreatment to posttreatment time period for both the affective disorders with personality problems patients and the affective disorders without personality problems patients. The estimation of bias/interactions showed the existence of specific behavioral differences between the two groups of patients. This study demonstrates the appropriateness of the methodological tool adopted, opening new expectations with regard to the integration of two approaches-psychoanalytical and pharmacological ones-in the treatment of psychiatric subjects.

Applying the many‐facet Rasch model to evaluate PowerPoint presentation performance in higher education

This study investigated the usefulness of the many‐facet Rasch model (MFRM) in evaluating the quality of performance related to PowerPoint presentations in higher education. The Rasch Model utilizes item response theory stating that the probability of a correct response to a test item/task depends largely on a single parameter, the ability of the person. MFRM extends this one‐parameter model to other facets of task difficulty, for example, rater severity, rating scale format, task difficulty levels. This paper specifically investigated presentation ability in terms of items/task difficulty and rater severity/leniency. First‐year science education students prepared and used the PowerPoint presentation software program during the autumn semester of the 2005–2006 school year in the ‘Introduction to the Teaching Profession’ course. The students were divided into six sub‐groups and each sub‐group was given an instructional topic, based on the content and objectives of the course, to prepare a PowerPoint presentation. Seven judges, including the course instructor, evaluated each group’s PowerPoint presentation performance using ‘A+ PowerPoint Rubric’. The results of this study show that the MFRM technique is a powerful tool for handling polytomous data in performance and peer assessment in higher education.

Developing a Measure of Therapist Adherence to Contingency Management: An Application of the Many-Facet Rasch Model

ABSTRACT A unique application of the Many-Facet Rasch Model (MFRM) is introduced as the preferred method for evaluating the psychometric properties of a measure of therapist adherence to Contingency Management (CM) treatment of adolescent substance use. The utility of psychometric methods based in Classical Test Theory was limited by complexities of the data, including: (1) ratings provided by multiple informants (i.e., youth, caregivers, and therapists), (2) data from separate research studies, (3) repeated measurements, (4) multiple versions of the questionnaire, and (5) missing data. Two dimensions of CM adherence were supported: adherence to Cognitive Behavioral components and adherence to monitoring components. The rating scale performed differently for items in these subscales, and of 11 items evaluated, eight were found to perform well. The MFRM is presented as a highly flexible approach that can be used to overcome the limitations of traditional methods in the development of adherence measures for evidence-based practices.

The Presence and Impact of Local Item Dependence on Objective Structured Clinical Examinations Scores and the Potential Use of the Polytomous, Many-Facet Rasch Model

Objective The purpose of this research project was to extend the research on the robustness of the dichotomous Rasch model to violations of the local independence assumption to the polytomous many-facet Rasch model (MFRM). Candidate scores from oral examinations and objective structured clinical examinations (OSCEs) have been shown to contain variance due to rater error/bias. If the MFRM is robust to local item dependence (LID), then the MFRM could theoretically be applied to medical OSCEs. Methods Five OSCEs were used in the study: 3 chiropractic licensure OSCEs and 2 nursing licensure OSCEs. Items were assigned to split-halves based on common stimulus. Split-half correlations were compared with Spearman-Brown estimates of reliability based on Cronbach α with all items contributing. Two- and 3-facet MFRM analyses were performed, first with individual items contributing and second with station totals contributing. Correlations were estimated between the 2 MFRM estimates. Results Cronbach α estimates with all items contributing were all very high (>.87). Spearman-Brown estimates were all considerably higher than split-half correlations. Correlations between MFRM by items and by stations were all very high (>.993). Conclusions The research project provided evidence that OSCEs violate the local item independence assumption. The project also showed that the MFRM is quite robust to such violations. The authors recommend that the MFRM be applied to OSCEs by station totals for estimates of candidate ability, and by items for item performance measures and quality control programs.

A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing

A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores. Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and (c) to make recommendations for improving the rating process. Variance due to Task (writing prompt) and the Person-by-Task interaction were high while the variance attributable to Raters and Occasion was low. Twenty-two percent of the variability in the ratings was unexplained. The common and unique features of generalizability theory and the Many-Facet Rasch Model are described, and the advantages and disadvantages of each are discussed.

Validity of a performance assessment of activities of daily living for people with developmental disabilities.

Since clients with different types of developmental disabilities often experience difficulties in activities of daily living (ADL), it is critical that assessments of ADL are evaluated in order to ensure that one can make valid judgements based on the results of the appraisal. The purpose of the present study was to evaluate the validity of a specific performance assessment instrument, the Assessment of Motor and Process Skills (AMPS), when used by occupational therapists with clients with developmental disabilities. Unlike global ADL assessments, the AMPS is used not only to evaluate the level of ADL dependence, but also to estimate the quality of each specific action performed when a person is performing ADL tasks. Data were gathered from 1724 participants with different developmental disabilities, including intellectual disability (ID), cerebral palsy and spina bifida. Many-Facet Rasch (MFR) analysis was used to examine person-response validity, and task and item scale validity. Goodness-of-fit statistics showed that the tasks and items had acceptable scale validity. The participants had acceptable person-response validity on the ADL motor scale, but had slightly lower than expected levels of person-response validity on the ADL process scale. The results indicate that clients with more severe forms of ID may have a higher proportion of different performance profiles in ADL than is expected by the MFR model of the AMPS. Since the proportion of participants who did not meet the criteria was only 3% lower than expected and in accordance with other studies, the difference may not be clinically meaningful. Otherwise, the results indicated that the AMPS is a valid tool when used with clients with developmental disabilities. Further research is needed to evaluate the use of the AMPS in clinical assessment and intervention planning for this group of clients.

論文評定データの解析における多相Raschモデルと分散分析モデルの比較

論文評定データの解析における多相Raschモデルと分散分析モデルの比較

'Home' Improvement: The Task for Measurement Specialists

This article challenges measurement specialists and other physical education and exercise science professionals to explore alternative models that are not based on classical test theory. Examples illustrate how the many-facet Rasch model has been used to solve measurement problems associated with performance assessments in the fields of medicine, business, and sport. A second challenge calls for more collaborative research efforts that address measurement problems in physical education and exercise science. The measurement subdiscipline will advance in the 21st century if these challenges are met.