Capsule Commentary on Post et al., Rating the Quality of Entrustable Professional Activities: Content Validation and Associations with the Clinical Context

Abstract

T he authors developed and validated a tool for assessing the quality of entrustable professional activities (QUEPA), and then evaluated 229 EPAs written by Mayo Clinic rotation directors. They reported higher ratings for inpatient than outpatient EPAs. EPAs based on interpersonal communication were rated highest, and those rating medical knowledge were lowest. This seems contrary to published findings suggesting that medical knowledge is easier to identify and evaluate than other competencies. This study is a necessary step in the implementation of EPA evaluation. The authors identified characteristics they felt EPAs should possess and evaluated the reliability of these characteristics. It is unclear why they deviated from those provided in the literature. The authors then used the tool themselves to rate new EPAs. The high reliability suggests strong agreement that EPAs should be focused, observable, and realistic/generalizable, and should integrate multiple competencies. As acknowledged by the authors, the major limitation is that their process and tool may not generalize to other contexts. The next challenge in EPA utilization is to go beyond developing reliable tools for evaluating EPAs, to developing reliable EPAs to evaluate residents. It has been suggested that EPAs, being based on trust, are different from learning objectives and better for evaluating competence, but this has not been demonstrated. In this study, PDs at the Mayo Clinic agreed on what makes a good EPA from an observer’s point of view, but it remains unclear whether EPAs are the best way to determine competence. The biggest challenge in using EPAs to evaluate competence is that we do not know the extent to which an observable behavior at any point in time is a good representation of an individual’s competence. As an analogy, behaviorism in psychology was abandoned because it limited understanding of the underlying mechanisms. Similarly, limited observations of even well-defined EPAs may be insufficient to understand where a resident is in their development of expertise. These limitations of the value of EPAs will need to be explored as we make the transition to outcomesbased assessments.