Surgical Skills Feedback and myTIPreport: Is There Construct Validity?

Abstract

To initiate construct validity testing of myTIPreport for procedural skill assessment in a prospective multicenter evaluation study. Teachers and learners from a convenience-based site selection of obstetrics and gynecology (OBGYN) and female pelvic medicine and reconstructive surgery (FPMRS) training programs performed procedural assessments in myTIPreport. The specifically defined 5-point Dreyfus rating scale describing ability levels from novice to expert was used. Defined as the degree to which a test or measure assesses what it was designed to measure, construct validity of myTIPreport was tested by comparing the medians of procedure-specific overall assessments, by both teachers and learners themselves, of senior learners with junior learners. To minimize type I error, comparisons were performed only when a threshold of 10 or greater feedback encounters per learner group was met. Correlation of teacher assessments and learner self-assessments was examined for myTIPreport. From November 2014 to May 2016, 12 OBGYN and 7 FPMRS training programs participated. There were 440 learners and 443 teachers. Feedback was recorded on 5,093 surgical procedures; 4,567 for OBGYN residents and 526 for FPMRS fellows. Each OBGYN procedure had two categories of teacher and learner assessments comparing postgraduate year (PGY)-4 with PGY-1 learner performance. This yielded 48 possible assessment comparisons for the included 24 OBGYN procedures. In all, 28 of these 48 (58%) met the threshold number of observations per learner group. In 28 of these 28 (100%) comparison categories, PGY-4s rated significantly higher than PGY-1s. Similarly, in 16 of 18 (89%) comparison categories meeting inclusion criteria, FPMRS PGY-7s rated significantly higher than FPMRS PGY-5s. Strong correlation was noted of teacher assessments and learner self-assessments in myTIPreport with a Spearman correlation coefficient of 0.89 (P<.001). As noted for the majority of compared teacher assessments and learner self-assessments, myTIPreport appeared to detect differences between senior and junior learners. These data support the emerging construct validity of myTIPreport for procedural skills assessment.