Components of Critical Decision Making and ABSITE Assessment: Toward a More Comprehensive Evaluation

Abstract

Accurate assessment of resident competency is a fundamental requisite to assure the training of physicians is adequate. In surgical disciplines, structured tests as well as ongoing evaluation by faculty are used for evaluating resident competency. Although structured tests evaluate content knowledge, faculty ratings are a better measure of how that knowledge is applied to real-world problems. In this study, we sought to explore the performance of surgical residents in a simulation exercise (strategic management simulations [SMS]) as an objective surrogate of real-world performance. Forty surgical residents participated in the SMS simulation that entailed decision making in a real-world-oriented task situation. The task requirements enable the assessment of decision making along several parameters of thinking under both crisis and noncrisis situations. Performance attributes include "simpler" measures of competency (activity level), intermediate categories (information management and emergency responses) to complex measures (breadth of approach and strategy). Scores obtained in the SMS were compared with the scores obtained on the American Board of Surgery In-Training Examination (ABSITE). The data were intercorrelated and subjected to a multiple regression analysis with ABSITE as the dependent variable and simulation scores as independent variables. Using a 1-tail test analysis, only 3 simulation variables correlated with performance on ABSITE at the .01 level (ie, basic activity, focused activity, task orientation). Other simulation variables showed no meaningful relationships to ABSITE scores at all. The more complex real-world-oriented decision-making parameters on SMS did not correlate with ABSITE scores. We believe that techniques such as the SMS, which focus on critical thinking, complement assessment of medical knowledge using ABSITE. The SMS technique provides an accurate measure of real-world performance and provides objective validation of faculty ratings.