The Development of an Independent Rater System to Assess Residentsʼ Competence in Invasive Procedures

Abstract

To design an independent rater (IR) direct observation system to monitor invasive procedures performed by residents in the hospital setting. The authors recruited, trained, and tested nonphysicians to become IRs for an Agency for Healthcare Research and Quality-funded study evaluating the impact of partial task simulation training of ultrasound-guided central venous catheter (CVC) insertion on skills transfer at a major academic medical center. IR applicants completed four hours of training: a two-hour didactic session and a two-hour testing session, including observation of 5 of 10 choreographed CVC insertion videotapes and completion of a 50-data-point procedural checklist. Eligibility to be hired as an IR included timing the procedure accurately, detecting technical errors and complications, and completing the procedural checklist accurately. Thirty-eight IR trainees completed the training module and videotape examinations. Twenty-seven (71%) trainees met criteria to be hired IRs by accurately assessing the duration of the procedure to within one minute, validating the checklist to within 95% accuracy, and detecting technical errors/complications to within a 3% margin of error. The authors found no association between educational level and hired status, and all 13 IRs assessed after the study had maintained their skills. Recent innovations in procedural training with partial task simulation trainers necessitate developing methods to measure skills transfer from the simulator to the clinical setting. This description of a nonphysician IR direct observation system for CVC insertion offers a feasible tool that may be generalized to monitoring other invasive procedures.