Objective Assessment Of Skills Research Articles

A PC-based laparoscopic surgery skills training and assessment system.

The purpose of this study is to build a cost-effective and easy-to-popularize laparoscopic training system based on improving traditional training box. The system has the capability of objective skills assessment and the function of automatic recording of training process and results, as well as 3-dimensional coordinate tracking of instruments. The results of pilot experiment in laparoscopic-assisted grip skill assessment had been shown the system can assess the different grip ability level between the senior surgeons and junior residents. Regarding to the evaluation of training effectiveness, five subjects without laparoscopic surgery experiences were asked to perform grip training for five days to observe their training curves. According to the experimental results, the total time taken for subject 1 to subject 5 are improved by 54.9%, 52.0%, 60.6%, 23.3%, and 63.5% separately.

Feasibility of a Low-Cost Instrumented Trocar for Universal Surgical Procedure Analyses1

The current path to become a surgeon follows the Halstedjan model of apprentice-style training, in which a resident remains under the shadow of a senior surgeon [1,2]. The resident starts by observing the surgeon and then gradually moves on to assist him. Though the process provides immediate mentor feedback through one-on-one interaction, it fails to carry out a systematic and consistent evaluation. All of the assessment is based on a senior surgeon’s subjective appraisal. Moreover, the requirement of performing a real procedure under the supervision of an experienced surgeon puts extra financial pressure on the medical school curriculum [2]. The advent and advancement of minimally invasive surgery (MIS) have made mastering such skills even more difficult, particularly psychomotor. Due to the increased complexity in required surgical skills and due to a need for a more consistent and systematic evaluation, there has been an increasing demand for an objective assessment of surgical skill. Studies in the past have examined motion analyses of surgeon’s hands and tool with a structured checklist to conclude that the motion analyses can serve as one of the measures for objectively assessing surgical skill [3]. A considerable amount of work has been done in the past to develop systems for tracking minimally invasive surgical tools. However, there does not exist a system that can be easily deployed to all three environments, i.e., virtual reality, physical benchtop, and operating room (OR). Most of these systems are only applicable to either virtual reality or physical benchtop training environment. There is only one system, Waseda bioinstrumentation 3, that can be applied in the OR. However, even that system is not validated yet and has an intrusive setup. Moreover, these systems can be very expensive with the average price ranging from $20,000 to $30,000. Hence, there is a strong need for a low-cost, noninvasive means to quantitatively discriminate surgical skill in a manner universally applicable to all environments and various surgical modalities like robotic, laparoscopic and endoscopic surgeries. This work presents the feasibility of such a system. Notably, we evaluate the feasibility of jerk cost as a low-cost alternative to skill evaluation. We hypothesize it can discriminate skill without expensive position tracking. 2 Methods

Determining the Role of a National Objective Assessment of Surgical Skills in Gynecological Oncology: An e-Delphi Methodology

The aim of this study is to determine a national consensus on the role of an objective assessment of technical surgical skills in gynecological oncology (GO). After approval was obtained from Society of Gynecologic Oncology of Canada, A panel of 20 GO leaders was assembled, representing all GO fellowship programs, and was asked to participate in an anonymous group and respond to an online 49-item questionnaire using a modified Delphi methodology. Ninety-five percent (n = 19) of those invited to participate did so. Seventeen of the panelists (89.5%) believed there was no sufficiently standardized technical skills assessment for GO fellows, whereas 18 responders (95%) believed that fellows should be objectively assessed on more than 1 occasion during their training. Consensus was predefined as Cronbach α greater than 0.8. The panel agreed on what procedures should be objectively assessed with a Cronbach α of 0.967. An overall Cronbach α of 0.993 was achieved after a single Delphi round. We achieved consensus on the possible components and logistics of a skills assessment process among a group of highly experienced GO trainers in Canada. This study provides the basis for further investigation and debate on the potential value, necessity, and feasibility of an assessment of advanced surgical and nonsurgical skills of GO trainees.

PROFICIENCY PERFORMANCE BENCHMARKS FOR REMOVAL OF SIMULATED BRAIN TUMORS USING NEUROTOUCH

BACKGROUND: Objective assessment of neurosurgical technical skills involved in the resection of cerebral tumors in operative environments is complex. Educators emphasize the need to develop and use objective and meaningful assessment tools that are reliable and valid for assessing trainees' progress in acquiring surgical skills. Novel technologies, such as virtual-reality simulation, have the potential to play important roles in the training of neurosurgeons. The purpose of this study was to develop benchmarks for a newly proposed set of objective measures (metrics) of neurosurgical technical skills performance during simulated brain tumor resection using a new virtual reality simulator (NeuroTouch). METHODS: A total of 31 participants were recruited including 16 ‘experts’ (neurosurgery staff) and 15 neurosurgery residents (‘novices’, 7 junior and 8 senior). Each participant performed 18 simulated brain tumor resections utilizing the NeuroTouch platform. The metrics for assessing surgical performance were computed using the NeuroTouch simulator and consisted of 1) Safety metrics including, volume of surrounding normal tissue removed, maximum force applied and sum of forces utilized during tumor resection 2) Quality of Operation metric which involved the percentage of tumor removed and 3) Efficiency metrics including duration for task completion, instrument path lengths and pedal activation frequency. RESULTS: The results demonstrated that ‘expert’ neurosurgeons (neurosurgery staff) resected less surrounding simulated normal brain tissue and less tumor tissue then residents. This data is consistent with the concept that ‘experts’ focused more on safety of the surgical procedure compared to novices. By analyzing experts' neurosurgical technical skills performance on these different metrics we were able to establish benchmarks for goal proficiency-based training of neurosurgery residents. CONCLUSIONS: Examining ‘expert’ neurosurgical performance in simulated settings such as NeuroTouch provides researchers with novel metrics for assessment of technical skills and development of proficiency based benchmarks. Identification of expert proficiency can led to improvements in resident training and assessment SECONDARY CATEGORY: n/a.

Low-cost assessment tools for the development of psychomotor competency in procedural surgical skills

Abstract. While didactic training is a crucial element of education in the health care profession, current technologies leveraging low cost data acquisition and processing may provide an attractive alternative means for rapid, objective assessment of the foundational skills. When these technologies are leveraged towards improving the procedural surgical skill set, there is a strong opportunity for enhancing current training practices. While the proctor will still play a crucial role in the refinement of clinical judgment, affordable options for rapid training of procedural skills may serve as an avenue to free an expert proctor to focus on the more nuanced elements of medical training. Here we propose and implement a system for the low cost assessment and training of procedural surgical skills. A knot-tying module and a tool articulation module were created using affordable sensing technologies, and initial validation with a trained clinician showed that the devices could discriminate correctly and poorly tied knots. The system lays the framework for a modular experiential component of skills development in the training of health professionals.

Identifying Successful Motor Task Completion via Motion-Based Performance Metrics

Objective assessment of skill is important in understanding the human performance of precision motor control tasks; however, outcome-based performance measures are most commonly used to provide feedback after the task is completed. In this paper, we propose the use of motion-based performance metrics in order to determine successful task completion strategies for a complex and unconstrained dynamic task. We developed a novel motor control task in a virtual environment and identified two motion-based metrics (mean absolute jerk and average frequency) that correlate with the successful performance of the task measured as a function of completion time and acquired targets. Our methodologies provide insight into the movement strategies used during successful trials of the motor control task, and could be used to provide specific feedback and coaching aimed at altering the task completion strategy and boosting the participants' performance.

Fine-motor skills testing and prediction of endovascular performance

Performing endovascular procedures requires good control of fine-motor digital movements and hand-eye coordination. Objective assessment of such skills is difficult. Trainees acquire control of catheter/wire movements at various paces. However, little is known to what extent talent plays for novice candidates at entry to practice. To study the association between performance in a novel aptitude test of fine-motor skills and performance in simulated procedures. The test was based on manual course-tracking using a proprietary hand-operated roller-bar device coupled to a personal computer with monitor view rotation. A total of 40 test repetitions were conducted separately with each hand. Test scores were correlated with simulator performance. Group A (n = 14), clinicians with various levels of endovascular experience, performed a simulated procedure of contralateral iliac artery stenting. Group B (n = 19), medical students, performed 10 repetitions of crossing a challenging aortic bifurcation in a simulator. The test score differed markedly between the individuals in both groups, in particular with the non-dominant hand. Group A: the test score with the non-dominant hand correlated significantly with simulator performance assessed with the global rating scale SAVE (R = -0.69, P = 0.007). There was no association observed from performances with the dominant hand. Group B: there was no significant association between the test score and endovascular skills acquisition neither with the dominant nor with the non-dominant hand. Clinicians with increasing levels of endovascular technical experience had developed good fine-motor control of the non-dominant hand, in particular, that was associated with good procedural performance in the simulator. The aptitude test did not predict endovascular skills acquisition among medical students, thus, cannot be suggested for selection of novice candidates. Procedural experience and practice probably supplant the influence of innate abilities (talent) over time.

THE ASIAN NUCLEAR MEDICINE BOARD (ANMB), WHY DO WE NEED IT?

THE ASIAN NUCLEAR MEDICINE BOARD (ANMB), WHY DO WE NEED IT?

Supplementation of surgical training using a model of objective skill assessment: “The par diagonal”

Supplementation of surgical training using a model of objective skill assessment: “The par diagonal”

The clinical impact of recognition and management of acutely ill surgical patient (ReMAP) course on FY1 assessment of an acutely unwell surgical patient

s / International Journal of Surgery 11 (2013) 686e745 700 ABSTRACTS predicts training opportunities and progression for surgical trainees. The conventional apprenticeship model in surgical education does not satisfy the requirements of training at present. We therefore introduced a model for objective skill assessment, the PAR-Diagonal Operating Matrix (PARDOM) which breaks down the task of vascular anastomosis into clearly defined skills. The PAR-DOM matrix provides a graphic representation of the progress of trainees and helps identify individual strengths and weaknesses. Using this model combined with 20 minutes a day out of hours exercise (performing needle skills on poached eggs, boiled potatoes and bananas) trainees were graphically seen to improve their PAR-DOM matrix scores within weeks. We therefore advocate the use of a PAR-DOM matrix scoring system by trainers and out of hours practice by trainees to counteract the reduction in theatre exposure and training opportunities. 1072: THE CLINICAL IMPACT OF RECOGNITION AND MANAGEMENT OF ACUTELY ILL SURGICAL PATIENT (REMAP) COURSE ON FY1 ASSESSMENT OF AN ACUTELY UNWELL SURGICAL PATIENT Jennifer Long, Hannah Vaughan-Williams, Joshua Moorhouse, Harsheet Sethi, Vidhi Misra, Nagappan Kumar. University Hospital Wales,

Open knot‐tying skills: Resident skills assessed

Open knot-tying and suturing skills are fundamental surgical skills, founding many alternative knot-tying techniques. It is therefore mandatory for residents to possess adequate basic open knot-tying skills. The aim of this study was to compare an objective assessment of open knot-tying skills by residents to a resident's own estimation of his or her knot-tying skills, before and after a knot-tying course. A prospective observational cohort study was performed. At baseline level, after 1 training day in the Advanced Suturing Course (ASC) in the Netherlands and Belgium and after 6 weeks of autonomous practice (i.e. self-practice), 99 residents' open knot-tying skills were objectively evaluated using the Objective Structured Assessment of Technical Skills (OSATS). The resident's own confidence in these skills was also evaluated. The ASC substantially and significantly improved residents' knot-tying skills according to the OSATS between baseline and post-measurement. The observed improvement after 1 training day decreased after 6 weeks of autonomous practice. Self-confidence increased directly after the training program and was maintained 6 weeks later. Residents having completed the first 3 years of residency displayed an overall greater self-confidence than residents not having completed the first 3 years of residency, although the increase in self-confidence was significantly larger in the latter after 6 weeks' autonomous training. There is a divergence between residents' objectified open knot-tying skills and self-confidence in these skills. The ASC improved open knot-tying skills according to the OSATS, however this improvement decreased after a 6-week period of autonomous practice. Self-confidence, in contrast, was maintained or increased. Further research is needed to correlate validated training programs with clinical outcomes and to determine whether residents' open knot-tying skills and self-confidence are retained beyond 1 year.

Recertification: What do specialists think about skill assessment?

BackgroundContinuing medical education and objective performance assessment remain the key components of recertification. Objective skills assessment in routine practice remains challenging due to extensive variations in case selection and treatments. This study explores expert opinions regarding objective skills assessment for specialists within the framework of recertification. MethodsWe used a qualitative, semi-structured interview-based approach to obtain information and suggestions about key issues and recommendations relating to specialists' skills assessment. Twenty-two face-to-face interviews were conducted. Interviews were transcribed and analysed by two reviewers. ResultsThe information from the interviews was categorized under the headings of: (1) the components of specialist-level skills, (2) the methods for assessing specialist skills, (3) the types of tools and procedures used during observational assessment, (4) the unsuccessful specialists, and (5) the selection and training of assessors for specialist assessment. ConclusionsOutcome-based assessment of performance followed by observation of practice, were recommended as effective modes of evaluation of performance.

22 A prospective randomized study of a new articulating laparoscopy device (Kymerax-Terumo) in the dry lab – part 1: Learning curve and objective skills assessment

22 A prospective randomized study of a new articulating laparoscopy device (Kymerax-Terumo) in the dry lab – part 1: Learning curve and objective skills assessment

Teaching Cricothyrotomy: A Multisensory Surgical Education Approach for Final-Year MedicalStudents

This study sought to evaluate the efficacy of a multisensory teaching approach in imparting the necessary knowledge, technical skills, and confidence to perform a cricothyrotomy to a cohort of fourth-year medical students. One hundred twenty students were recruited into the study. Subjects commenced by viewing an upper airway obstruction vodcast. Students subsequently observed an expert demonstration of cricothyrotomy on human cadavers. Following this, students were able to practice on cadaveric simulators guided by experts. The students' confidence was assessed with pre- and post-session confidence-level questionnaires. Thirty randomly selected students were then assessed with a post-session objective skill assessment test. Finally, a multiple choice examination was administered to test knowledge. Ninety-five percent of student subjects achieved a score of at least 80% on the knowledge examination. All sampled students were considered competent to perform cricothyrotomy as judged by 2independent expert observers using the objective skill assessment test instrument. There was a statistically significant improvement between the pre- and post-CLQ scores (p<0.001). In keeping with current education theories on multisensory learning, qualitative feedback suggested a student preference for this teaching approach. This study provides further evidence that a multisensory teaching intervention effectively improves the knowledge, skill, and confidence of fourth-year medical students in performing cricothyrotomy.

Toward an objective assessment of technical skills: a national survey of surgical program directors in Saudi Arabia.

BackgroundAfter almost a decade of implementing competency-based programs in postgraduate training programs, the assessment of technical skills remains more subjective than objective. National data on the assessment of technical skills during surgical training are lacking. We conducted this study to document the assessment tools for technical skills currently used in different surgical specialties, their relationship with remediation, the recommended tools from the program directors’ perspective, and program directors’ attitudes toward the available objective tools to assess technical skills.MethodsThis study was a cross-sectional survey of surgical program directors (PDs). The survey was initially developed using a focus group and was then sent to 116 PDs. The survey contains demographic information about the program, the objective assessment tools used, and the reason for not using assessment tools. The last section discusses the recommended tools to be used from the PDs’ perspective and the PDs’ attitude and motivation to apply these tools in each program. The associations between the responses to the assessment questions and remediation were statistically evaluated.ResultsSeventy-one (61%) participants responded. Of the respondents, 59% mentioned using only nonstandardized, subjective, direct observation for technical skills assessment. Sixty percent use only summative evaluation, whereas 15% perform only formative evaluations of their residents, and the remaining 22% conduct both summative and formative evaluations of their residents’ technical skills. Operative portfolios are kept by 53% of programs. The percentage of programs with mechanisms for remediation is 29% (19 of 65).ConclusionThe survey showed that surgical training programs use different tools to assess surgical skills competency. Having a clear remediation mechanism was highly associated with reporting remediation, which reflects the capability to detect struggling residents. Surgical training leadership should invest more in standardizing the assessment of surgical skills.

Objective Skills Assessment and Construct Validation of a Virtual Reality Temporal Bone Simulator

To evaluate construct validity of the Voxelman TempoSurg Virtual Reality (VR) temporal bone simulator by determining whether generated objective metrics can distinguish experienced otologic surgeons from intermediate and novice surgeons. Prospective assessment study. Two university-affiliated teaching hospitals. Sixty-five participants were recruited; 40 novice surgeons, 15 trainees in otolaryngology, and 10 experienced otolaryngology consultants with a specialist interest in otology were individually assessed on a standardized simulated temporal bone task. The task involved identification and delineation of the sigmoid sinus in a virtual left-sided temporal bone. Objective data were produced using a scoring matrix incorporated into the VOXEL MAN TempoSurg software. The simulator measured the total time taken to complete the task, the volume and efficiency of bone removal and error data for excessive force or injury to the facial nerve, dura, and sigmoid sinus. Experts and intermediates outperformed novices with respect to the total time taken to complete the task (expert versus novice: p < 0.001; intermediate versus novice: p < 0.001), total volume of bone removed (p < 0.001 and p = 0.03), efficiency of bone removal (p < 0.001 and p < 0.001), time spent with the drill tip obscured (p = 0.002 and p < 0.001), and number of injuries to the sigmoid sinus (p < 0.001 and p < 0.001). The intermediate group injured the sigmoid sinus on more occasions than the experts (p = 0.008) and were less efficient than experienced surgeons (p = 0.005). Simulator-generated objective metrics can be used to differentiate individuals of differing levels of experience using a standardized temporal bone task. VR simulation has potential as a training tool and may have a role in both formative and summative assessment.

Objective assessment in residency‐based training for transoral robotic surgery

To develop a robotic surgery training regimen integrating objective skill assessment for otolaryngology and head and neck surgery trainees consisting of training modules of increasing complexity leading up to procedure-specific training. In particular, we investigated applications of such a training approach for surgical extirpation of oropharyngeal tumors via a transoral approach using the da Vinci robotic system. Prospective blinded data collection and objective evaluation (Objective Structured Assessment of Technical Skills [OSATS]) of three distinct phases using the da Vinci robotic surgical system in an academic university medical engineering/computer science laboratory setting. Between September 2010 and July 2011, eight otolaryngology-head and neck surgery residents and four staff experts from an academic hospital participated in three distinct phases of robotic surgery training involving 1) robotic platform operational skills, 2) set up of the patient side system, and 3) a complete ex vivo surgical extirpation of an oropharyngeal tumor located in the base of tongue. Trainees performed multiple (four) approximately equally spaced training sessions in each stage of the training. In addition to trainees, baseline performance data were obtained for the experts. Each surgical stage was documented with motion and event data captured from the application programming interfaces of the da Vinci system, as well as separate video cameras as appropriate. All data were assessed using automated skill measures of task efficiency and correlated with structured assessment (OSATS and similar Likert scale) from three experts to assess expert and trainee differences and compute automated and expert assessed learning curves. Our data show that such training results in an improved didactic robotic knowledge base and improved clinical efficiency with respect to the set up and console manipulation. Experts (e.g., average OSATS, 25; standard deviation [SD], 3.1; module 1, suturing) and trainees (average OSATS, 15.9; SD, 3.9; week 1) are well separated at the beginning of the training, and the separation reduces significantly (expert average OSATS, 27.6; SD, 2.7; trainee average OSATS, 24.2; SD, 6.8; module 3) at the conclusion of the training. Learning curves in each of the three stages show diminishing differences between the experts and trainees, which is also consistent with expert assessment. Subjective assessment by experts verified the clinical utility of the module 3 surgical environment, and a survey of trainees consistently rated the curriculum as very useful in progression to human operating room assistance. Structured curricular robotic surgery training with objective assessment promises to reduce the overhead for mentors, allow detailed assessment of human-machine interface skills, and create customized training models for individualized training. This preliminary study verifies the utility of such training in improving human-machine operations skills (module 1), and operating room and surgical skills (modules 2 and 3). In contrast to current coarse measures of total operating time and subjective assessment of error for short mass training sessions, these methods may allow individual tasks to be removed from the trainee regimen when skill levels are within the standard deviation of the experts for these tasks, which can greatly enhance overall efficiency of the training regimen and allow time for additional and more complex training to be incorporated in the same time frame.

A curriculum to teach and evaluate resident skills in the management of postpartum hemorrhage

The aim of this study was the development of a curriculum for the management of postpartum hemorrhage and an objective assessment of technical skills (OSATS) for uterine compression sutures. Twenty-two residents participated in the study. Evaluations included the global rating scale, task-specific checklist, and pass/fail rating for the OSATS and task-specific checklist for the hemorrhage drill. The reliability and validity of the evaluation tools were calculated. The inter-rater reliability was 0.98 for the hemorrhage drill checklist, 0.77 for the global rating scale, and 0.93 for the uterine suture checklist. Construct validity evaluation revealed senior residents performed superiorly to junior residents on the global rating scale (P=0.006) and on the uterine suture checklist (P=0.04). There was an improvement in performance on the post-test in comparison to the pretest (P=0.001). We present an inexpensive, reliable, and valid curriculum to teach and evaluate the medical and surgical management of postpartum hemorrhage.

Measuring Clinical Decision Making

Reliable and objective assessment of clinical decision-making skills has been a long-standing goal in occupational testing in the allied health professions. With this goal in mind, the key features problem (KFP) format was developed which elicits targeted decisions about key features of clinical scenarios. To build on a small body of empirical evidence evaluating their efficacy, this study evaluates whether KFPs successfully assess higher order cognitive processes. Analysis of objective data (item length and difficulty, item performance, and response times) and subjective data (expert ratings of cognitive complexity) supported the proposition that KFPs tend to be more cognitively complex than conventional multiple-choice questions. Not only were they rated as more complex, but this complexity accounted for some of the increase in time spent responding to these items. Results support the use of KFPs in standardized assessments for measuring higher order cognitive processes such as clinical decision making.

Virtual temporal bone dissection system: OSU virtual temporal bone system

AbstractObjectives/Hypothesis:The objective of this project was to develop a virtual temporal bone dissection system that would provide an enhanced educational experience for the training of otologic surgeons.Study Design:A randomized, controlled, multi‐institutional, single‐blinded validation study.Methods:The project encompassed four areas of emphasis: structural data acquisition, integration of the system, dissemination of the system, and validation.Results:Structural acquisition was performed on multiple imaging platforms. Integration achieved a cost‐effective system. Dissemination was achieved on different levels including casual interest, downloading of software, and full involvement in development and validation studies. A validation study was performed at eight different training institutions across the country using a two‐arm randomized trial where study subjects were randomized to a 2‐week practice session using either the virtual temporal bone or standard cadaveric temporal bones. Eighty subjects were enrolled and randomized to one of the two treatment arms; 65 completed the study. There was no difference between the two groups using a blinded rating tool to assess performance after training.Conclusions:A virtual temporal bone dissection system has been developed and compared to cadaveric temporal bones for practice using a multicenter trial. There was no statistical difference between practice on the current simulator compared to practice on human cadaveric temporal bones. Further refinements in structural acquisition and interface design have been identified, which can be implemented prior to full incorporation into training programs and used for objective skills assessment.