The Development and Validation of the Diabetic Wound Assessment Learning Tool

Abstract

Diabetic foot wounds are estimated to result in one major lower extremity amputation globally every 30 seconds, in addition to their economic and societal costs. Physicians and trainees encounter these patients in all specialties, particularly in medical and emergency departments. Few studies focus on how to teach and assess wound management. Given the importance of “assessment for learning” in competence-based medical education, we aimed to develop, and examine specific sources of validity evidence for, an assessment tool of wound management competencies. We organized our tool development and validation process using Kane’s framework. First, we used a nominal group process involving nine Canadian experts in diabetic wound management to develop the tool items and two 10-minute simulation-based testing scenarios. We then collected validity evidence for the tool by assessing 74 participants’ (61 physicians, 13 nonphysicians) performance during the two scenarios: 44 novices (<50 previous cases), 17 intermediates (50-500), and 13 experts (>500). Two assessors independently rated participants using our tool. We evaluated reliability using Generalizability Theory, and measured test-retest reliability with the intraclass correlation coefficient (ICC) comparing performance during each scenario. We also compared performance scores across the three experience levels using analysis of variance. Cronbach’s alpha was 0.95 for the tool, implying high internal consistency. Test-retest reliability was also excellent (ICC, 0.94; 95% confidence interval [CI], 0.91-0.96) for single measures and (ICC, 0.97; 95% CI, 0.95-0.98) for average measures. Pooled inter-rater reliability was fair for single measures (ICC, 0.71; 95% CI, 0.68-0.73) and good for average measures (ICC, 0.83; 95% CI, 0.81-0.85). The tool differentiated between novices and the other two groups well (P < .01), but not between intermediates and experts (P = .34). Our generalizability coefficient was 0.87 and our Phi coefficient was 0.87. Our generalizability analyses suggest that scores on the Diabetic Wound Assessment Learning Tool can safely be generalized over the prepared testing scenarios and can be used to provide absolute ratings as well as relative ratings of wound management skills. The accumulated validity evidence suggests our tool can reliably be used to assess novice clinicians’ competence in diabetic wound management during simulated cases.