Procedure-based Assessment Research Articles

Quality of Feedback Using Workplace Based Assessments in Urological Training

Introduction: The ISCP (Intercollegiate Surgical Curriculum Programme) was introduced in 2007. It heralded a shift in surgical training from the traditional apprenticeship model, to a system where competence is assessed regularly within the workplace. Central to ISCP is the recording of Workplace Based Assessments, such as PBA (procedure-based assessment) and DOPS (Direct Observation of Procedural Skills). These are assessments “for” learning rather than assessments “of” learning and central to their purpose is detailed feedback from trainer to trainee. The aim of this study was to assess the quality of the feedback recorded on ISCP. Methods: The anonymised ISCP portfolios for 170 core and specialty urology trainees working in 16 deaneries were analysed. Over a three month period 469 DOPS and 592 PBAs were recorded and the presence and quality of assessors' feedback were evaluated according to the presence of five criteria: • Encouragement through positive language, • Including strengths, • Raising appropriate development needs, • Suggestions for appropriate corrective action, • Providing explanations. Results: Feedback was recorded in 49% and 67% of PBAs and DOPS respectively. In those assessments where feedback was recorded, positive language was used in 83% in PBAs and 70% in DOPS. Strengths were recorded in 53% for both tools. Developmental needs, suggestions for development and detailed explanation were recorded in 25%, 26%, 29% for PBAs and 24%, 23%, 19% for DOPS respectively. Conclusion: This study highlights the lack of feedback being recorded on ISCP and the paucity of good quality feedback when it was given.

Procedure-based assessments: an evaluation of their role in reflective learning

Procedure-based assessment (PBA) has, together with other common workplace assessments, become the cornerstone of a surgical trainee's evaluation process, the annual review of competency progression.

Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

Objectives To compare user satisfaction and acceptability, reliability and validity of three different methods of assessing the surgical skills of trainees by direct observation in the operating theatre across a range of different surgical specialties and index procedures. Design and setting A 2-year prospective, observational study in the operating theatres of three teaching hospitals in Sheffield. Methods The assessment methods were procedure-based assessment (PBA), Objective Structured Assessment of Technical Skills (OSATS) and Non-technical Skills for Surgeons (NOTSS). The specialties were obstetrics and gynaecology (O&G) and upper gastrointestinal, colorectal, cardiac, vascular and orthopaedic surgery. Two to four typical index procedures were selected from each specialty. Surgical trainees were directly observed performing typical index procedures and assessed using a combination of two of the three methods (OSATS or PBA and NOTSS for O&G, PBA and NOTSS for the other specialties) by the consultant clinical supervisor for the case and the anaesthetist and/or scrub nurse, as well as one or more independent assessors from the research team. Outcome measures Information on user satisfaction and acceptability of each assessment method from both assessor and trainee perspectives was obtained from structured questionnaires. The reliability of each method was measured using generalisability theory. Aspects of validity included the internal structure of each tool and correlation between tools, construct validity, predictive validity, interprocedural differences, the effect of assessor designation and the effect of assessment on performance. Results Of the 558 patients who were consented, a total of 437 (78%) cases were included in the study: 51 consultant clinical supervisors, 56 anaesthetists, 39 nurses, 2 surgical care practitioners and 4 independent assessors provided 1635 assessments on 85 trainees undertaking the 437 cases. A total of 749 PBAs, 695 NOTSS and 191 OSATSs were performed. Non-O&G clinical supervisors and trainees provided mixed, but predominantly positive, responses about a range of applications of PBA. Most felt that PBA was important in surgical education, and would use it again in the future and did not feel that it added time to the operating list. The overall satisfaction of O&G clinical supervisors and trainees with OSATS was not as high, and a majority of those who used both preferred PBA. A majority of anaesthetists and nurses felt that NOTSS allowed them to rate interpersonal skills (communication, teamwork and leadership) more easily than cognitive skills (situation awareness and decision-making), that it had formative value and that it was a valuable adjunct to the assessment of technical skills. PBA demonstrated high reliability (G > 0.8 for only three assessor judgements on the same index procedure). OSATS had lower reliability (G > 0.8 for five assessor judgements on the same index procedure). Both were less reliable on a mix of procedures because of strong procedure-specific factors. A direct comparison of PBA between O&G and non-O&G cases showed a striking difference in reliability. Within O&G, a good level of reliability (G > 0.8) could not be obtained using a feasible number of assessments. Conversely, the reliability within non-O&G cases was exceptionally high, with only two assessor judgements being required. The reasons for this difference probably include the more summative purpose of assessment in O&G and the much higher proportion of O&G trainees in this study with training concerns (42% vs 4%). The reliability of NOTSS was lower than that for PBA. Reliability for the same procedure (G > 0.8) required six assessor judgements. However, as procedure-specific factors exerted a lesser influence on NOTSS, reliability on a mix of procedures could be achieved using only eight assessor judgements. NOTSS also demonstrated a valid internal structure. The strongest correlations between NOTSS and PBA or OSATS were in the ‘decision-making’ domain. PBA and NOTSS showed better construct validity than OSATS, the year of training and the number of recent index procedures performed being significant independent predictors of performance. There was little variation in scoring between different procedures or different designations of assessor. Conclusions The results suggest that PBA is a reliable and acceptable method of assessing surgical skills, with good construct validity. Specialties that use OSATS may wish to consider changing the design or switching to PBA. Whatever workplace-based assessment method is used, the purpose, timing and frequency of assessment require detailed guidance. NOTSS is a promising tool for the assessment of non-technical skills, and surgical specialties may wish to consider its inclusion in their assessment framework. Further research is required into the use of health-care professionals other than consultant surgeons to assess trainees, the relationship between performance and experience, the educational impact of assessment and the additional value of video recording. Funding The National Institute for Health Research Health Technology Assessment programme.

Training in the MMC Era: Procedure-Based Assessments – the Parents' Perspective

The recent implementation of Modernising Medical Careers (MMC) has had a significant impact on the way that both medical and surgical trainees in the UK are trained and clinically or technically assessed. The aim of MMC is 'to drive up the quality of care for patients through reform and improvement in postgraduate medical education and training'. Surgical training has also been affected by the final stage of the implementation of the European Working Time Regulations. One of the primary changes in MMC has been the introduction of a formal assessment system of the trainees. The new system was initiated to progress towards more competency-based training rather than the total time spent in training. The trainee's progress in achieving clinical and technical competencies is assessed, as is the quality of the trainee and the training he or she receives.

The fallacy of comparing surgeons with pilots in the search for safer surgical training

Standards and methods for the assessment of surgical skills are under increased scrutiny as a result of public and professional reactions to clinical failures1. Concerns are that surgeons have a false perception of their abilities, leading to avoidable deaths. To prevent this, better surgical skills assessment has been proposed1. Comparisons between surgery and aviation as two hazardous industries have led to the recommendation of mandatory simulation training before patient encounters2, but the disciplines of aviation and surgery are not strictly comparable. In contrast to aviation simulators, surgical simulators are incomplete to equip a surgeon with sufficient skills to take directly to a real-life operating theatre. The complexity and variation in human anatomy, physiology and disease inevitably means that a fully standardized approach cannot apply to every operation. The skills needed for a non-standardized surgical environment are different from those of a standardized aviation environment. Although surgeons and pilots share a thinking process to determine the correct action by interpreting behavioural clues, the necessary coordination in aviation is explicit, resulting from preset instructions in a highly standardized environment3. Some assumptions about simulated surgical skills training and translation into clinical practice do not stand up to close examination. The autopilot has become a standard component in large aeroplanes. The first fully automated transatlantic flight took place in 1947. Autopilot is the only way to control aeroplanes in many flight phases owing to the effect of temperature on aeroplane parts and surrounding air turbulence4. Simulation is the only way to train commercial pilots in the absence of real flight training opportunities in the air. Robotic simulation is often portrayed as an analogous device for the surgeon, but there is no form of autosurgeon. Robots available for surgical procedures are simple slave mechanical devices that work only with direct human interaction. Another belief is that practice on real patients is dangerous. There is no evidence that supervised surgical training compromises patient safety. Teaching hospitals have better or comparable outcomes to nonteaching hospitals after case-mix correction5. It is 26 times safer to fly than to drive a car6, so perhaps it might be more reasonable to compare surgery with driving. It is still standard to learn driving with an instructor on the road. Drivers, like surgeons, operate in a minimally standardized environment with many variations. Drivers have an agreed thinking process to avoid hazards faced, but no step-by-step instructions. Surgeons have to deal with anatomical and pathological variations, along with unexpected events in the operating theatre, in the same way that drivers have to respond to unexpected events on the road. Driving test assessment tools provide further insight into this similarity. In many countries, the driving test examines both theoretical knowledge and practical skill7. In the UK, the theoretical part consists of multiple choice questions (MCQ) to assess knowledge and hazard perception clips to check learners’ awareness, followed by a period of instructorfacilitated training on the roadwhen it is illegal for the learner to drive alone. Once the learner is fully trained, the practical test is taken. This involves an on-road driving assessment with an examiner armed with a checklist. The clinical skills literature supports the relevance of this type of assessment for the development of surgical skills. Miller8 provided a pyramidal structure to assess clinical skills, with lower levels about knowledge covered by MCQ. Surgical training and assessment are performed in the operating theatre by the trainer using procedure-based assessment forms9. This covers the remaining practical skills assessment levels, with clear parallels to the practical driving test. In surgical training, hazard perception might be achieved with error recovery training10. This stresses subjective detection of errors committed by a third party while using content to justify own errors. Dror10 suggested the use of interactive training clips to detect errors in others, linking error detection with the search for a recovery plan. Making these links should replace self-error justification and denial mechanisms. This new cognitive process could reduce errors and mitigate damage during operations. Errors are infrequent in the standardized automated

Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology

To compare user satisfaction and acceptability, reliability and validity of three different methods of assessing the surgical skills of trainees by direct observation in the operating theatre across a range of different surgical specialties and index procedures. A 2-year prospective, observational study in the operating theatres of three teaching hospitals in Sheffield. The assessment methods were procedure-based assessment (PBA), Objective Structured Assessment of Technical Skills (OSATS) and Non-technical Skills for Surgeons (NOTSS). The specialties were obstetrics and gynaecology (O&G) and upper gastrointestinal, colorectal, cardiac, vascular and orthopaedic surgery. Two to four typical index procedures were selected from each specialty. Surgical trainees were directly observed performing typical index procedures and assessed using a combination of two of the three methods (OSATS or PBA and NOTSS for O&G, PBA and NOTSS for the other specialties) by the consultant clinical supervisor for the case and the anaesthetist and/or scrub nurse, as well as one or more independent assessors from the research team. Information on user satisfaction and acceptability of each assessment method from both assessor and trainee perspectives was obtained from structured questionnaires. The reliability of each method was measured using generalisability theory. Aspects of validity included the internal structure of each tool and correlation between tools, construct validity, predictive validity, interprocedural differences, the effect of assessor designation and the effect of assessment on performance. Of the 558 patients who were consented, a total of 437 (78%) cases were included in the study: 51 consultant clinical supervisors, 56 anaesthetists, 39 nurses, 2 surgical care practitioners and 4 independent assessors provided 1635 assessments on 85 trainees undertaking the 437 cases. A total of 749 PBAs, 695 NOTSS and 191 OSATSs were performed. Non-O&G clinical supervisors and trainees provided mixed, but predominantly positive, responses about a range of applications of PBA. Most felt that PBA was important in surgical education, and would use it again in the future and did not feel that it added time to the operating list. The overall satisfaction of O&G clinical supervisors and trainees with OSATS was not as high, and a majority of those who used both preferred PBA. A majority of anaesthetists and nurses felt that NOTSS allowed them to rate interpersonal skills (communication, teamwork and leadership) more easily than cognitive skills (situation awareness and decision-making), that it had formative value and that it was a valuable adjunct to the assessment of technical skills. PBA demonstrated high reliability (G > 0.8 for only three assessor judgements on the same index procedure). OSATS had lower reliability (G > 0.8 for five assessor judgements on the same index procedure). Both were less reliable on a mix of procedures because of strong procedure-specific factors. A direct comparison of PBA between O&G and non-O&G cases showed a striking difference in reliability. Within O&G, a good level of reliability (G > 0.8) could not be obtained using a feasible number of assessments. Conversely, the reliability within non-O&G cases was exceptionally high, with only two assessor judgements being required. The reasons for this difference probably include the more summative purpose of assessment in O&G and the much higher proportion of O&G trainees in this study with training concerns (42% vs 4%). The reliability of NOTSS was lower than that for PBA. Reliability for the same procedure (G > 0.8) required six assessor judgements. However, as procedure-specific factors exerted a lesser influence on NOTSS, reliability on a mix of procedures could be achieved using only eight assessor judgements. NOTSS also demonstrated a valid internal structure. The strongest correlations between NOTSS and PBA or OSATS were in the 'decision-making' domain. PBA and NOTSS showed better construct validity than OSATS, the year of training and the number of recent index procedures performed being significant independent predictors of performance. There was little variation in scoring between different procedures or different designations of assessor. The results suggest that PBA is a reliable and acceptable method of assessing surgical skills, with good construct validity. Specialties that use OSATS may wish to consider changing the design or switching to PBA. Whatever workplace-based assessment method is used, the purpose, timing and frequency of assessment require detailed guidance. NOTSS is a promising tool for the assessment of non-technical skills, and surgical specialties may wish to consider its inclusion in their assessment framework. Further research is required into the use of health-care professionals other than consultant surgeons to assess trainees, the relationship between performance and experience, the educational impact of assessment and the additional value of video recording.

Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre

Procedure-based assessment (PBA) is used within most UK surgical training programmes for assessing trainees' procedural skills in the operating theatre. All postgraduate assessment methods require evidence to support their implementation. The aims were to evaluate the validity, reliability and acceptability of PBA. Eighty-one trainees in six surgical specialties were assessed performing common procedures; 749 PBAs were provided across 348 operations by 57 clinical supervisors and four independent assessors. Construct validity was demonstrated by correlation of PBA scores with measures of surgical training and experience. Reliability (G exceeding 0·8) of the adjusted total item score and global summary score for a given procedure was achieved using four and three assessor judgements respectively; assessing a mix of procedures required more cases/assessors because performance is procedure specific. The acceptability of PBA for assessment and feedback within surgical training was predominantly positive among clinical supervisors and trainees. PBA demonstrated good overall validity and acceptability, and exceptionally high reliability. Trainees should be assessed adequately for each given procedure.

Surgical Procedure-based Assessment: How Did We Get Here and Where are we Going?

The Intercollegiate Surgical Curriculum Programme (ISCP) has implemented a tool for assessment of surgical technique called procedure-based assessment (PBA). It is part of a spectrum of workplace-based assessments being used for the surgical specialist training programmes. We reviewed the literature on observational assessment of surgical practice in order to analyse how PBAs relate to the evidence base for this form of assessment.

Undertaking Procedure-Based Assessment is Feasible in Clinical Practice

With the development of a new curriculum, workplace based assessments such as procedure-based assessment (PBA) are becoming increasingly common within surgical training. However, there have been concerns about the impact of these assessments on clinical practice. This study assessed the time taken to complete PBA forms to determine whether it is feasible in clinical practice. PBAs for three colorectal procedures (anterior resection, right hemicolectomy and anal fistula) were undertaken by various trainers and trainees. A pilot study was performed to identify potential reasons for incomplete forms and procedural modifications subsequently applied in the main study. Times taken to complete the consenting and operative components of the forms were recorded. Incomplete forms in the pilot were mainly attributable to time constraints. In the main study, all assessments were completed within 30 min. Assessment times increased with complexity of the procedure. Median times for completing the consenting and operative components in anterior resection were 13 min (range, 8-15 min) and 15 min (range, 10-18 min), respectively. PBAs are feasible in clinical practice and are valued by trainees as a means of enabling focused feedback and targeted training. Commitment from trainers and trainees will be required but, with adequate planning, the assessment tool is effective with minimal impact on clinical practice.

Workplace‐based assessment: assessing technical skill throughout the continuum of surgical training

The Royal Colleges of Surgeons and Surgical Specialty Associations in the UK have introduced competence-based syllabi and curricula for surgical training. The syllabi of the Intercollegiate Surgical Curriculum Programme (ISCP) and Orthopaedic Curriculum and Assessment Programme (OCAP) define the core competencies, that is, the observable and measureable behaviours required of a surgical trainee. The curricula define when, where and how these will be assessed. Procedure-based assessment (PBA) has been adopted as the principal method of assessing surgical skills. It combines competencies specific to the procedure with generic competencies such as safe handling of instruments. It covers the entire procedure, including preoperative and postoperative planning. A global summary of the level at which the trainee performed the assessed elements of the procedure is also included. The form has been designed to be completed quickly by the assessor (clinical supervisor) and fed-back to the trainee between operations. PBA forms have been developed for all index procedures in all surgical specialties. The forms are intended to be used as frequently as possible when performing index procedures, as their primary aim is to aid learning. At the end of a training placement the aggregated PBA forms, together with the logbook, enable the Educational Supervisor and/or Programme Director to make a summary judgement about the competence of a trainee to perform index procedures to a given standard.

Transferring simulated arthroscopic skills to the operating theatre

The aim of this study was to investigate the effect of laboratory-based simulator training on the ability of surgical trainees to perform diagnostic arthroscopy of the knee. A total of 20 junior orthopaedic trainees were randomised to receive either a fixed protocol of arthroscopic simulator training on a bench-top knee simulator or no additional training. Motion analysis was used to assess performance objectively. Each trainee then received traditional instruction and demonstrations of diagnostic arthroscopy of the knee in theatre before performing the procedure under the supervision of a blinded consultant trainer. Their performance was assessed using a procedure-based assessment from the Orthopaedic Competence Assessment Project and a five-point global rating assessment scale. In theatre the simulator-trained group performed significantly better than the untrained group using the Orthopaedic Competence Assessment Project score (p = 0.0007) and assessment by the global rating scale (p = 0.0011), demonstrating the transfer of psychomotor skills from simulator training to arthroscopy in the operating theatre. This has implications for the planning of future training curricula.

Assessment of performance in orthopaedic training

The reliable measurement of performance is a problem faced by training authorities worldwide. At a recent international orthopaedic conference, none of a group of experienced orthopaedic educators could report significant progress on this issue. The current Record of In Training Assessment (RITA)