CUSUM analysis to assess competence: what failure rate is acceptable?

Abstract

Cumulative sum (CUSUM) analysis is used to assess competency in practical procedures. Previous studies have used CUSUM to assess competence in administering labour epidural, but these assessments were mainly performed with novice trainees. Different acceptable failure rates for the same procedure, such as administering labour epidural, have been used. In this article, we describe our experience of using CUSUM for the continuing assessment of competence among experienced trainees. We also tested the hypothesis that setting different failure rates can lead to different conclusions about a trainee's competency. After obtaining departmental approval, we collected the records of the first 50 labour epidurals performed by trainees at specialty training (ST) 3/4 level, and plotted two CUSUM graphs for each trainee. For the first CUSUM graph, we used 10 and 20 per cent as acceptable and unacceptable failure rates, respectively. In the second CUSUM graph, we used 15 and 30 per cent as acceptable and unacceptable failure rates, respectively. We analysed the CUSUM data of six trainees. With 10 and 20 per cent as acceptable and unacceptable failure rates, only four trainees demonstrated competence. The median number of attempts required was 25, with a range of 25-32 attempts. However, with 15 and 30 per cent used as the acceptable and unacceptable failure rates, five out of six trainees demonstrated competence, with a median of 18 and range of 14-33 attempts. Our study shows that CUSUM analysis can be successfully used for the continued assessment of competence in experienced trainees. It also shows that setting different failure rates can produce different results, leading to confusion and inconsistency when comparing CUSUM results. With the increasing use of CUSUM analysis, we believe that the organisations responsible for training, such as the Royal College of Anaesthetists, should provide guidance on what is an acceptable failure rate for different procedures, such as the administration of an epidural during labour, at different stages of training.