Abstract
The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand. We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons. The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments. We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
