The Reliability of a Musculoskeletal Objective Structured Clinical Examination in a Professional Physical Therapist Program

Abstract

Background and Purpose. The Objective Structured Clinical Examination (OSCE) incorporates case scenarios, uniform grading schemes, and standardized patients to assess students' clinical competence. Constructing an OSCE that has good reliability can be a challenge, especially if the outcome of the exam is to make a pass/fail decision. Similar to studies reported in the literature, this study determined the interrater reliability and internal consistency of an OSCE used to examine professional physical therapist students' competencies for musculoskeletal conditions.Participants. Sixty-five doctor of physical therapy (DPT) degree students from 2 universities were recruited to participate as subjects in this study.Methods. The musculoskeletal OSCE (mOSCE) was developed for testing students' cognitive and psychomotor skills by assessing their ability to examine, evaluate, and provide interventions for peripheral joint musculoskeletal conditions based on clinical experience and evidence-based practice. During a 4-year time period, the score sheet and case scenarios were modified based on feedback from experienced clinicians and faculty culminating in a pilot study that laid the foundation for this research project. The mOSCE was based on 2 case scenarios consisting of 7 stations, using detailed score sheets to score each student. The score sheet allowed the examiners to evaluate the technical performance and decision-making processes used by students when determining the appropriate orthopedic clinical skills to implement.Results. The mOSCE investigated in this study was a mid-term assessment of the students' clinical competency after completing half of the curriculum in the musculoskeletal tract of the program. Thus, a reliability coefficient of ≥ 0.60 and ≤ 0.90 was considered acceptable for both inter-rater reliability and internal consistency. Interrater reliability among examiners reached acceptable reliability coefficients for the total score given in the exam for all students (ICC = 0.77; program A students: ICC = .73, program ? students: ICC = 0.70); however, as expected, internal consistency values were poor and did not reach the acceptable threshold.Discussion and Conclusion. Interrater reliability and internal consistency are 2 factors that contribute to a reliable evaluation tool. The process of developing an examination that is reliable in examining student competencies can be a daunting task, especially when these examinations are often used to base pass/fail decisions. The primary outcome of this research project established evidence of an OSCE that has good interrater reliability and poor internal consistency. A secondary outcome of the project demonstrated improved reliability coefficients from those found in our pilot study, which was conducted prior to this report in response to improvements made in examiner training.Key Words: Teaching methods, Entry-level education, Evaluation research, Orthopedics/ musculoskeletal content.INTRODUCTIONThe Objective Structured Clinical Examination (OSCE) was initially described in the medical literature by Harden and Gleeson1 in 1975 and has since expanded into other health professional programs, including physical therapy. The OSCE has been used in evaluating students' abilities to take a history, perform a physical examination, counsel patients, or provide patient management.14 In addition, the purpose of the OSCE has expanded to include the assessment of students' clinical competencies related to course objectives for a course examination and students' clinical competencies for board and licensure examinations.5'6Whether the OSCE is used within a course or for a high-stake examination such as licensure, the format of the OSCE is highly structured incorporating case scenarios, uniform grading schemes, and standardized patients who interact with the students during the exam. First, the OSCE utilizes case scenarios to test specific competencies across a series of stations. …