Accrediting Competence in Colonoscopy: Validity and Reliability of the UK Joint Advisory Group/NHS Bowel Cancer Screening Programme Accreditation Assessment

Abstract

After national UK audit demonstrated very variable quality of colonoscopy, the introduction of the NHS Bowel Cancer Screening Programme prompted the introduction of a searching assessment to select the screening programme's future colonoscopists. Leading endoscopists, trainers and the JAG collaborated to develop an appropriate assessment tool after consultation with clinical educationalists in the UK, USA, and Holland. This comprised performance data, a knowledge test (MCQ), and a DOPS (Direct Observation of Procedural Skills) assessment. The aims were to determine the validity and reliability of the assessment. A true/false 30 item x 5 answer MCQ was devised covering key issues. To ensure both professional and educational input, DOPS forms from two national training centres were reviewed by the Trainers Group appointed under the NHS National Endoscopy Training Programme, and important colonoscopy skills were identified and amalgamated into domains. Detailed descriptors for all skill levels were written for each domain. An outline curriculum was published, and assessors underwent both the assessment and detailed assessor training. Performance and demographic data, as well as assessment data both from the MCQ and DOPS over two cases by two assessors were collected prospectively from candidates. Semi-structured questionnaires were collected from candidates and assessors on the validity of the assessment. Colonoscopists submitted performance data and attempted the assessment. In more than 3000 paired judgments on over 100 candidates during more than 200 cases, there was 96% congruence of agreement across the pass/fail divide (98% for major domains), and between 60-83% absolute congruence across individual domains between assessors, e.g. demonstrates awareness of patient's consciousness and pain 73%; adequate mucosal visualisation 81%. An expert global opinion agreed with the grading system in 97% of cases. The grading correlated weakly with self-reported caecal intubation rates and MCQ scores (r = 0.24 & 0.27, p < 0.01) respectively. No correlation was seen between the grading and colonoscopy numbers in previous year. Of the candidates 93% felt the DOPS assessment was fair or very fair overall, whilst 86% felt the MCQ was fair or very fair. Of the assessors, 75% felt the DOPS was valid or very valid, others felt it was not testing enough nor reliably assessing therapy, whilst 100% of them overall felt the process was fair or very fair. The DOPS accreditation has good face and content validity, and high reliability across the crucial pass/fail divide. Work is in place improving the MCQ.