Validation of an Assessment Tool for OSCE “Fill in the Gap” Milestone on Pelvic Mass

Abstract

OBJECTIVE: To develop a validated, observed structured clinical exam assessment tool for evaluation of the pelvic mass milestone (level 1–3) for OB/GYN resident learners. METHODS: Competency-based, validated assessment tools in many of the ACGME milestone areas for OB/GYN are lacking, so we proposed administering an OSCE to supplement existing tools. Four ACGME milestones (pelvic mass, surgical skills, stress urinary incontinence, and preconception care) were selected for evaluation based on: 1) the paucity of existing data for our resident learners on which to base milestone achievement or progression in these areas; and 2) perceived difficulty in obtaining consistent and robust individual assessment of these milestones during clinical care. Content experts developed OSCE scenarios and assessment tools locally. Each assessment tool was comprised of multiple clinical skills (elements) from milestone levels 1–3 for evaluation of the learner. Milestone level 1 learners demonstrate milestone expected of an incoming resident, level 2 learners are advancing but are not yet performing at mid-residency level, and level 3 learners are consistently demonstrating the majority of milestones targeted for residency. Level 4 and 5 milestones were not assessed, as they require more robust, multi-dimensional evaluations. Each element was assigned a milestone level corresponding to the ACGME milestone descriptors. The pelvic mass scenario and assessment tool were internally and externally validated using the modified Delphi technique with three faculty over three rounds. An expert panel consisting of three specialist OB/GYN faculty members involved in graduate medical education was asked to review the milestone elements and corresponding levels for content validity eliciting data from learner to patient. The panel also determined how many elements per milestone level a competent, independent practicing physician (defined as having achieved at least milestone level 4 or graduation target) would be expected to achieve during a routine clinical encounter. Learners were required to achieve 70% or more elements to pass the milestone level. Data were analyzed with descriptive statistics. EVALUATION: Thirty-six out of 36 residents performed the OSCE in the spring of 2016, participating in four separate clinical scenarios. Two faculty completed blinded evaluations on each resident. There were 58 elements assessed for the pelvic mass scenario: 18 elements from L1, 22 elements from L2, and 11 elements for L3. Five elements (one from L2, and four from L3) were achieved by only 5% of learners. Outcome measures included milestone achievement as defined above. The majority of residents were able to achieve a passing score for milestones level 1 and 2 compared with level 3 (71.9% versus 31%). Compared with junior residents, senior residents were able to achieve a passing score on at least one or more milestone (64% versus 50%), and were more likely to pass all three milestones (67% versus 33%), demonstrating construct validity. Only 11% of resident learners did not achieve a passing score on any of the milestone levels, and these residents were 5 times more likely to be junior level residents versus senior level residents, again demonstrating construct validity. DISSEMINATION: Abstract will be submitted for presentation at the 2017 CREOG and APGO annual meeting, followed by manuscript preparation and peer-reviewed submission to AAMC MedEdPORTAL. REFLECTIVE CRITIQUE: OSCE-based pelvic mass milestone assessment is a valid tool for evaluation of resident learners with discriminatory capability between junior and senior level learners. However, the process of developing OSCE scenarios is time-consuming, and administering an OSCE can be costly and requires significant faculty participation. For this particular project, a preliminary needs assessment, internal and external review of content, and the ability to discriminate junior from senior level performance following administration of the OSCE enhances the validity of this assessment tool. Milestone elements that were only achieved by 1–2 learners, or no learners, are considered elements that require reevaluation to enhance future OSCE assessments.