A preliminary study on the reliability and validity of objective structured skills assessment for the or-thopeadic postgraduates

Abstract

Objectives To design an objective and structured evaluation system for the clinical competence of orthopedic postgraduates in the diagnosis and treatment of distal radius fractures, and to ana-lyze its reliability and validity. Methods 28 orthopaedic postgraduates representing six levels of surgical training were tested for competence in performing surgical approach for distal radius fracture on cadaver spec-imens during which four measures were used to assess competency: examination of basic theory based on net-work item bank, objective structured operation assessment,overall assessment and operation examination re-sults. In addition, the time for completion of the surgery was also recorded. Each assessment tool was correlat-ed with the others as well as with the resident’s level of training. Results There was a significant correla-tion between the seniority of candidates and the score of theoretical examination (F=6.193, P=0.000), the score of structured operation examination (F=6.374, P=0.002), the score of overall assessment (F=2.321, P=0.030), and the passing rate of final operation examination (F=36.300, P=0.000). No significant differ-ences were found between seniority and time to completion of the surgical approach exposure (F=2.282, P<0.073). Conclusions The results of the present study suggested that both theoretical examination and cadaver testing discriminate between novice and accomplished postgraduates. However, although the theo-retical test scores could predict the operational test results, but the theoretical results can not guarantee excellent oper-ational skills. Key words: Orthopeadic postgraduates; Objective structured skills assessment; Reliability and validity analysis