Use of Ecological Momentary Assessment to Guide Curricular Change in Graduate Medical Education

Abstract

To assess whether a novel evaluation tool could guide curricular change in an internal medicine residency program. The authors developed an 8-item Ecological Momentary Assessment tool and collected daily evaluations from residents of the relative educational value of 3 differing ambulatory morning report formats (scale: 8 = best, 0 = worst). From the evaluations, they made a targeted curricular change and used the tool to assess its impact. Residents completed 1388 evaluation cards for 223 sessions over 32months, with a response rate of 75.3%. At baseline, there was a decline in perceived educational value with advancing postgraduate (PGY) year for the overall mean score (PGY-1, 7.4; PGY-2, 7.2; PGY-3, 7.0; P < .01) and for percentage reporting greater than 2 new things learned (PGY-1, 77%; PGY-2, 66%; PGY-3, 50%; P < .001). The authors replaced the format of a lower scoring session with one of higher cognitive content to target upper-level residents. The new session's mean score improved (7.1 to 7.4; P = .03); the adjusted odds ratios before and after the change for percentage answering, "Yes, definitely" to "Area I need to improve" was 2.53 (95% confidence interval [CI], 1.45-4.42; P = .001) and to "Would recommend to others," it was 2.08 (95% CI, 1.12-3.89; P = .05). The Ecological Momentary Assessment tool successfully guided ambulatory morning report curricular changes and confirmed successful curricular impact. Ecological Momentary Assessment concepts of multiple, frequent, timely evaluations can be successfully applied in residency curriculum redesign.