Global Rating Scores Research Articles

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting

BackgroundCompetence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.MethodsThe study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.ResultsValidity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.ConclusionsWe designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

Laparoscopic Surgical Skills are Significantly Improved by the Use of a Portable Laparoscopic Simulator: Results of a Randomized Controlled Trial

There is growing evidence that training on virtual reality simulators leads to improved performance in the animate and human operating room. However, they are expensive, have a limited availability, and involve complex systems. Portable simulators are significantly cheaper, more user-friendly, and are flexible systems that are more suited to a surgical trainee's busy schedule. The use of portable surgical simulators to train skills and reduce errors has never been evaluated in prospective, randomized clinical settings. The objective of this study was to determine if training on the portable Integrated Laparoscopic Simulator leads to improved performance of core laparoscopic skills. Core laparoscopic skills were identified by five experienced laparoscopic surgeons and modeled into two exercises and three basic tasks. Twenty surgically naive medical students had baseline laparoscopic skills assessed on a fixed simulator. Participants were randomized to either 14 h training on a portable laparoscopic simulator over a 3 week period, or control with no training. At 3 weeks two expert laparoscopic surgeons blinded to the allocation of participants assessed their pre- and post-intervention performances recorded on a CD-ROM. The outcome measures included time to complete and global rating scores of clipping and dissection tasks. No differences were observed in baseline skills level between the two groups. The intervention group had better quality of scissors dissection (p = 0.0038) and improved clipping skills (p = 0.0051), and they took less time to accomplish the tasks (p = 0.0099) in comparison to control. Training on the portable Integrated Laparoscopic Simulator significantly improved core laparoscopic skills in medical students with no prior experience.

Trauma’12 Winning Abstract - End Product Assessment in Microsurgery Simulation Training - A New Insider View

Background There are various assessment methods that allow the evaluation of skills acquisition in microsurgery simulation training such as global rating scores and check lists. The present scoring systems are used to analyse the process of conducting an anastomosis. This paper presents a novel quantitative method based on assessing the intimal surface of the anastomosis to analyse the end product. Method 24 candidates were recruited for a five day microsurgical skills acquisition course. The cohort ranged from undergraduate medical students to experts microsurgeons. On days three and five, the trainees performed two assessment anastomoses on cryopreserved rat aortas (average diameter = 1.75 millimeter). These were cut open to expose the intimal surface of the anastomosis and a high magnification photo was taken. The photos were then analysed using ImageJ. The perfect anastomosis was defined as having equidistant insertion points, equal suture lengths across the anastomosis line, and a single axis meaning that there is no torsion in the vessel edges. In turn, 4 parameters were measured: 1) distance between the proximal insertion points, 2) distance between the distal insertion points, 3) length of sutures placed, 4) number of axes. Using these parameters, a 10 component scoring system was produced by a 3 point linkert scale giving 0, 5, and 10 points for scores between the range of 0–50th, 51st to 75th and above the 75th centile respectively. The maximum score achievable is 100. Results The thresholds of this scoring system were defined based on the population performance. In our cohort the scores ranged from 0–95. This system demonstrated both concurrent and construct validity (p Conclusion This novel scoring system satisfies an area of microsurgical skills acquisition assessment that has not yet been covered in great depth. The system allows objective assessment of the anastomosis's quality thus paving the way for identifying clinical safe threshold of microsurgery simulated training. In the future we can correlate this system to physiological anastomosis outcome measures such as patency rate and other methods of microsurgery assessment skills to establish the predictive validity of this system in discriminating trainees who have reached clinically competent standards in microsurgery.

LAPAROSCOPIC SURGICAL TRAINING. A THREE STAGES MODEL

LAPAROSCOPIC SURGICAL TRAINING. A THREE STAGES MODEL (Abstract): The classic “Halstedian” apprenticeship model for surgical training takes place into the operating theater under the strict coordination of a senior surgeon. The development of minimally invasive surgery (MIS) rapidly revealed that this apprenticeship model is insufficient to successful fulfill the well known three stages of training of Rasmussen model – skill-based behavior (SBB), rulebased behavior (RBB) and knowledge-based behavior (KBB) – due to ethical, medico-legal and economic considerations and a specific training using different methods (theoretical and hands-on courses using simulators, animal models and web resources) is mandatory. However it isn’t a worldwide accepted laparoscopic training curriculum. We present our experience with different types of teaching methods; in this way we retrospectively reviewed our experience in laparoscopic education, presenting data using Rasmussen model, as a dynamic three stages model: 1) first stage (the beginning), from 1993 until 1995, our staff acquired SBB, RBB and KBB for basic MIS procedures: cholecystectomy, appendectomy, surgical ablation of ovarian cysts; 2) the second stage, from 1996 until 1999, characterized by continuous training of our surgical staff until the achievement of the KBB level for basic MIS procedures and training for other operations (laparoscopic treatment of the groin hernias, eso-gastric and bariatric surgery) and the teaching of the residents and the surgeons from Moldova region to achieve SBB and RBB for basic MIS procedures; 3) the third stage, from 2000 until present characterized by a continuous improvement of the surgical teams’ laparoscopic skills for different procedures, and developing a modern curricula for laparoscopic education. CONCLUSIONS: There is no universal method for laparoscopic training. Our experience demonstrates good results using a combination of training methods which allowed acquiring: SBB (Basic skills) using virtual reality (VR) simulators and box trainers; RBB (especially for laparoscopic cholecystectomy) using multimedia resources, live operations / proctorship, VR simulators (with force feed-back), box trainers (liver-gallbladder porcine model) and live pig surgery; KBB using VR simulators, multimedia resources, live operation / proctorship, and live pig surgery. An assessment method for laparoscopic training is mandatory; in this way we prefer reviewing the recording data by experts using Global Rating Score (GRS).

Chronic Graft-Versus-Host Disease (cGVHD) Patients with Bronchiolitis Obliterans Syndrome (BOS) Have Worse Clinical Manifestations and Severity of cGVHD with More Impairment in Self-Assessed Physical and Mental Health than cGVHD Patients without BOS

AbstractAbstract 1939 Background:One of the most severe manifestations of cGVHD is bronchiolitis obliterans syndrome (BOS), an insidious lung disease occurring in a subset of patients after allogeneic HSCT, characterized by progressive circumferential fibrosis and cicatrization of the small terminal airways. Since BOS is morbid and frequently fatal complication, insight into the features that distinguish those patients with cGVHD who develop BOS from those who do not could be important in better understanding of the disease and its clinical implications, and may lead to developing of new treatment modalities and strategies. The aim of this study was to compare demographic characteristics, clinical manifestations, laboratory parameters, functional and psychological status, activity and severity of cGVHD, and overall survival (OS) between cGVHD patients with and without BOS. Methods:46 adult cGVHD patients with BOS (diagnosed using modified NIH criteria as: FEV1<75% and a) FEV1/FVC ratio <0.7 or b) air trapping defined by both RV or RV/TLC and CT criteria) and 165 adult cGVHD patients without BOS were enrolled from 2004–2011 into the cross sectional NCI cGVHD natural history study. A detailed clinical, laboratory, and functional evaluation was performed, specifically obtaining: NIH organ system severity scoring, PFTs, chest CT, cGVHD activity, intensity of immunosuppression, number of prior systemic therapies (PST), cGVHD global rating score reported by clinician and patient (form A and B), SF36 (PCS, MCS), Lee symptom scale (total and breathing subscale), HAP (MAS, AAS), 2-minute walk velocity, and grip strength. The association between these factors and BOS was initially evaluated using univariate methods to test for a difference, and then multivariable logistic regression for prediction. Median follow-up of survivors was 47 months (range 10–90). Results:cGVHD patients with and without BOS were not different regarding age, gender, primary diagnosis, conditioning intensity, donor gender and cell source, time from transplant to enrollment or to cGVHD diagnosis, median number of PST for cGVHD, history of aGVHD, cGVHD onset and classification, intensity of immunosuppression, and cGVHD activity. cGVHD patients with BOS had fewer HLA matched donors (p=0.0044), worse cGVHD NIH global severity (p<0.0001), worse NIH lung (p<0.0001) and eye (p=0.011) scores, worse respiratory symptoms (p<0.0001), worse cGVHD global rating score by clinician (p=0.0038) and patient (p=0.0089), lower Karnofsky (p=0.0001), longer time from cGVHD diagnosis to enrollment (0.032), lower Schirmer’s test (p=0.029), lower walk velocity (p=0.0084), lower HAP MAS and AAS (p<0.0001) scores, more breathing symptoms (p<0.0001) and greater fatigue (p=0.04). Those with BOS also reported inferior physical (p=0.008) and mental (p=0.006) health. Having BOS was also associated with higher platelet count (p=0.0089), WBC (p=0.028), ANC (p=0.046), LDL (p=0.0029), and IgA (p=0.032) comparing to cGVHD patients without BOS. In a predictive statistical model higher platelet count, higher RV and higher lung function score (LFS) can correctly predict 84.8% cGVHD patients with BOS and 80.4% without BOS. BOS did not significantly affect OS; however, in univariate analyses, BOS patients with FEV1<50% had significantly shorter OS compared to patients without BOS (p<0.0001) and compared to those who had BOS with FEV1≥50% (p=0.0006). A 4 year survival was 95% for patients with BOS and FEV1>=50, 78% for patients without BOS, and 37% for patients with BOS and FEV1< 50%, respectively. Conclusion:cGVHD patients with BOS had worse clinical manifestations and cGVHD severity as well as inferiorities in functioning and self-assessed physical and mental health. The impact of BOS on survival was restricted to an effect of FEV1<50% among those with BOS. Disclosures:No relevant conflicts of interest to declare.

Evaluating the correlation between the HUESAD and OSATS scores: Concurrent validity study

Background: The aim of this study was to determine whether a correlation exists between the HUESAD and both the OSATS checklist and the global rating score. Material and methods: Fourteen participants were recruited, ranging from residents to consultants. The HUESAD measured three different parameters: Visual spatial ability, smoothness and accuracy. The OSTAS measured the endoscopic surgical skills by the checklist and the global rating score. Results: A significant relationship was thus identified between the OSATS checklist (p < 0.0001), the OSATS global rating score (p < 0.05) and the HUESAD assessment in the evaluation of visual-spatial ability. In terms of smoothness, there was also a significant relationship between the OSATS checklist (p < 0.05), the OSATS global rating score (p < 0.05) and the HUESAD assessment. Moreover, there was a significant relationship between the OSATS checklist (p < 0.05), the OSATS global rating score (p < 0.05) and the HUESAD assessment for accuracy. Conclusions: There was strong correlation between the motion analysis in the HUESAD assessment and OSATS checklist and the global rating score. Moreover, the visual spatial ability, smoothness and accuracy analyzed by the HUESAD were reliable parameters when assessing the endoscopic surgical skills.

Telephone referrals by junior doctors: a randomised controlled trial assessing the impact of SBAR in a simulated setting

ObjectiveTo determine whether exposing junior doctors to Situation, Background, Assessment, Recommendation (SBAR) improves their telephone referrals. SBAR is a standardised minimum information communication tool.MethodsA randomised controlled trial with participants and...

The Impact of Stress on Paramedic Performance During Simulated Critical Events

Substantial research demonstrates that the stressors accompanying the profession of paramedicine can lead to mental health concerns. In contrast, little is known about the effects of stress on paramedics' ability to care for patients during stressful events. In this study, we examined paramedics' acute stress responses and performance during simulated high-stress scenarios. Twenty-two advanced care paramedics participated in simulated low-stress and high-stress clinical scenarios. The paramedics provided salivary cortisol samples and completed an anxiety questionnaire at baseline and following each scenario. Clinical performance was videotaped and scored on a checklist of specific actions and a global rating of performance. The paramedics also completed patient care documentation following each scenario. The paramedics demonstrated greater increases in anxiety (P < .05) and salivary cortisol levels (P < .05) in response to the high-stress scenario compared to the low-stress scenario. Global rating scores were significantly lower in the high-stress scenario than in the low-stress scenario (P < .05). Checklist scores were not significantly different between the two scenarios (P = .12). There were more errors of commission (reporting information not present in the scenario) in the patient care documentation following the high-stress scenario than following the low-stress scenario (P < .05). In contrast, there were no differences in omission errors (failing to recall information present in the scenario) between the two scenarios (P = .34). Clinical performance and documentation appear vulnerable to the impact of acute stress. This highlights the importance of developing systems and training interventions aimed at supporting and preparing emergency workers who face acute stressors as part of their every day work responsibilities.

Nursing Care, Inpatient Satisfaction, and Value-Based Purchasing

The objective of this study was to investigate how domains of patient satisfaction in hospitals predict Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) global rating scores to support hospital administrators in dealing with reimbursement changes. Medicare's inpatient value-based purchasing program ties a portion of hospital payments to scores on the HCAHPS survey. Logistical regression analysis of patient-level satisfaction domain scores from a standard, commercially available survey and HCAHPS top-box scores controlled for potential confounding variables. Each 1-point increase in the nursing domain score increased the odds of achieving an HCAHPS top-box score by 4.9%. Increases in other domain scores also affected top-box odds, although to a smaller extent. Hospital administrators wishing to maximize Medicare reimbursement will realize the greatest impact by improving patient satisfaction with nursing care.

PMO-220 Patient anticipation of some pain goes along way when predicting overall satisfaction with a colonoscopy procedure

IntroductionProviding a quality patient experience is a key facet of the Global Rating Score (GRS). Patient surveys are considered an integral means of assessing satisfaction. Meeting the patient's expectations is...

A Comparison of 2 Ex Vivo Training Curricula for Advanced Laparoscopic Skills

To compare the effectiveness and cost of 2 ex vivo training curricula for laparoscopic suturing. Although simulators have been developed to teach laparoscopic suturing, a barrier to their wide implementation in training programs is a lack of knowledge regarding their relative training benefit and their associated cost. This prospective single-blinded randomized trial allocated 24 surgical residents to train to proficiency using either a virtual reality (VR) simulator or box trainer. All residents then placed intracorporeal laparoscopic stitches during a Nissen fundoplication on a patient. The operating room (OR) cases were video-recorded and technical proficiency was assessed using 2 validated tools. OR performance of both groups was compared to that of conventionally trained residents and to fellowship-trained surgeons. A cost analysis of box training, VR training, and conventional residency training across Canadian surgical programs was performed. After ex vivo training, no significant differences in laparoscopic suturing in the OR were found between the 2 groups with respect to time (P = 0.74)-global rating score (P = 0.65) or checklist score (P = 0.97). It took conventionally trained residents 6 practice attempts in the OR to achieve the technical proficiency of the ex vivo trained groups (P = 0.83). VR training was more efficient than box training (transfer effectiveness ratio of 2.31 vs 1.13). The annual cost of training 5 residents on the FLS trainer box was $11,975.00, on the VR simulator was $77,500.00, and conventional residency training was $17,380.00. Over 5 years, box training was the most cost-effective option for all programs, and VR training was more cost-effective for programs with more 10 residents. Training on either a VR simulator or on a box trainer significantly decreased the learning curve necessary to learn laparoscopic suturing. VR training, however, is the more efficient training modality, whereas box training the more cost-effective option.

Simulating Various Levels of Clinical Challenge in the Assessment of Clinical Procedure Competence

Immersive simulation is increasingly used for competency assessment of emergency physicians. This group's concept of hybrid simulation (HS) (combining simulated patients and part-task trainers (a simulator that simulates a limited component of a clinical procedure) to create a multimodal clinical context) requires clinicians to integrate technical and nontechnical skills in a holistic clinical performance for assessment. It also offers the potential to provide authentic simulation of a given clinical procedure across multiple levels of challenge. The aims of this study are to systematically design and validate 2 patient-focused HS scenarios (each combining a simulated patient with a part-task simulator) for assessment of the management of a commonly encountered problem in an emergency department (ED) at different levels of clinical challenge, and to explore the effect of level of challenge of the HS scenario on physicians' performance. A simple (HS1) and a complex (HS2) HS scenario (based on the management of a patient with a traumatic skin laceration within the ED) was developed according to expert opinion through cognitive task analysis. Interns and emergency medicine residents (stratified into expert and novice groups according to experience) were recruited to participate in both scenarios. Participants were randomized to perform either the HS1 or HS2 scenario first. Participants completed a questionnaire for face validity (realism of simulation) and content validity (comprehensiveness of simulation). Performances were assessed by 2 independent raters using validated rating tools modified to the needs of this study: the Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, the Objective Structured Assessment of Technical Skills-Global Rating Score, and the Direct Observation of Procedural Skills. Ten novice and 10 expert clinicians completed both scenarios. Mean face and content validity ratings were high for both HS1 (mean 4.4 [SD 0.52] and 4.2 [SD 0.48], respectively) and HS2 scenarios (mean 4.5 [SD 0.35] and 4.3 [SD 0.43], respectively). In HS1, no difference was found between experts' and novices' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, Objective Structured Assessment of Technical Skills-Global Rating Score, and Direct Observation of Procedural Skills ratings. Experts performed significantly better than novices in HS2 in terms of the 3 tools' ratings. Novices' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist and Direct Observation of Procedural Skills ratings were significantly worse in HS2 compared with HS1, but no difference was found with the Objective Structured Assessment of Technical Skills-Global Rating Score. No statistical difference was found in experts' Modified Objective Structured Assessment of Technical Skills-Task Specific Checklist, Objective Structured Assessment of Technical Skills-Global Rating Score, and Direct Observation of Procedural Skills ratings between HS2 and HS1 scenarios. Recreating clinical challenge is an important consideration in the design of simulation-based assessment of procedural skills of clinicians. In this study, we have demonstrated a systematic approach to developing HS scenarios, which may be able to recreate various levels of clinical challenge for purpose of assessment of procedural skills.

1514 LAPAROSCOPIC SURGICAL SKILLS ARE SIGNIFICANTLY IMPROVED BY THE USE OF A PORTABLE LAPAROSCOPIC SIMULATOR RESULTS OF A RANDOMISED CONTROLLED TRIAL

You have accessJournal of UrologyTechnology & Instruments: Surgical Education & Skills Assessment/Ureteroscopy II1 Apr 20121514 LAPAROSCOPIC SURGICAL SKILLS ARE SIGNIFICANTLY IMPROVED BY THE USE OF A PORTABLE LAPAROSCOPIC SIMULATOR RESULTS OF A RANDOMISED CONTROLLED TRIAL Thomas Johnston, Jean Ker, Benjie Tang, and Ghulam Nabi Thomas JohnstonThomas Johnston Dundee, United Kingdom More articles by this author , Jean KerJean Ker Dundee, United Kingdom More articles by this author , Benjie TangBenjie Tang Dundee, United Kingdom More articles by this author , and Ghulam NabiGhulam Nabi Dundee, United Kingdom More articles by this author View All Author Informationhttps://doi.org/10.1016/j.juro.2012.02.1281AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookTwitterLinked InEmail INTRODUCTION AND OBJECTIVES There is growing evidence that training on virtual reality simulators leads to an improved performance in the animate and human operating room. However, they are expensive, have a limited availability and involve complex systems. Portable simulators are significantly cheaper, more user-friendly, and are flexible systems which more suit a surgical trainee's busy schedule. The use of portable surgical simulators to train skills and reduce errors has never been evaluated in prospective, randomised clinical settings. The objective of this study was to determine if training on the portable Integrated Laparoscopic Simulator leads to an improved performance of core laparoscopic skills. METHODS Core laparoscopic skills were identified by five experienced laparoscopic surgeons and modelled into exercises. Twenty surgical naive medical students had baseline laparoscopic skills assessed on a fixed simulator. Participants were randomised to either 14 hours training on a portable laparoscopic simulator over a three week period, or control with no training. At 3 weeks two expert laparoscopic surgeons blinded to the allocation of participants assessed their pre and post intervention performances recorded on a CD-ROM using validated OSAT score sheet. The primary outcomes were outcome measures included time to complete and global rating scores of a clipping and dissection tasks. RESULTS No differences were observed in baseline skills level between the two groups. The intervention group had better quality of scissor dissection (p value 0.0038); improved clipping skills (p value 0.0051) and took less time to accomplish the tasks (p value 0.0099) in comparison to control (Table 1). Table 1 Post intervention laparoscopic skills and OSAT scores in both the groups Laparoscopic Skills Intervention group Mean (SD) Control group Mean (SD) Statistical significance P value (95% Confidence Interval) Scissors handling and dissection (OSATS score) 20.7(4.08) 11.70(7.12) 0.0038 (3.43-14.56) Time taken for dissection task (seconds) 116.90(52.02) 221.50(74.49) 0.0019 (−164.96 to-44.24) Cutting tasks(OSATS score) 20.80(3.49) 11.70(7.67) 0.0051 (14.90-−3.29) Time taken for cutting task (seconds) 54.9(14.27) 92.90(35.91) 0.0061 (−63.68 to-12.32) CONCLUSIONS Training on the portable Integrated Laparoscopic Simulator significantly improved core laparoscopic skills in medical students with no prior experience. © 2012 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 187Issue 4SApril 2012Page: e613 Advertisement Copyright & Permissions© 2012 by American Urological Association Education and Research, Inc.MetricsAuthor Information Thomas Johnston Dundee, United Kingdom More articles by this author Jean Ker Dundee, United Kingdom More articles by this author Benjie Tang Dundee, United Kingdom More articles by this author Ghulam Nabi Dundee, United Kingdom More articles by this author Expand All Advertisement Advertisement PDF downloadLoading ...

Impact of stress on resident performance in simulated trauma scenarios

Training and practice in medicine are inherently stressful. The effects of stress on performance in clinical situations are poorly understood. The purpose of this study was to examine the stress responses and clinical performance of residents during low and high stress (HS) simulated trauma resuscitations. Thirteen emergency medicine and general surgery residents were evaluated in HS and low stress (LS) trauma resuscitation simulations. Subjective and physiologic (heart rate, salivary cortisol) responses were measured at baseline and in response to the scenarios. Performance was assessed with global rating and checklist scores of technical performance, time to record critical information, and the Anesthesia Non-Technical Skills tool. Post-scenario recall was assessed with the completion of a standardized trauma history form. Post-scenario subjective stress and cortisol levels were higher in the HS scenario compared with the LS scenario (p < 0.05). Checklist performance scores and post-scenario recall were significantly lower in the HS compared with the LS condition (p < 0.05). In trainees, some aspects of performance and immediate recall appear to be impaired in complex clinical scenarios in which they exhibit elevated subjective and physiologic stress responses. The findings of this study highlight a potential threat to patient safety and demand further investigation. Future studies should strive to further elucidate the effects of stress on specific components of performance and investigate ways to reduce its negative impact.

Objective Structured Assessment of Technical Skill in Upper Extremity Surgery

Objective assessment of technical skills in hand surgery has been lacking. This article reports on an Objective Structured Assessment of Technical Skills format of a multiple bench-station evaluation of orthopedic surgery residents' technical skills for 3 common upper extremity surgeries. Twenty-seven residents (6 postgraduate year [PGY] 2, 8 PGY 3, 8 PGY 4, and 5 PGY 5) participated in the examination. Each resident performed surgery on a cadaveric specimen at 3 stations, trigger finger release (TFR), open carpal tunnel release, and distal radius fracture fixation. A board-certified hand surgeon evaluated trainee performance at each station, using a procedure-specific detailed checklist, a validated global rating scale, and pass/fail assessment. A resident post-testing evaluation was collected. Construct validity with correlation between year in training and detailed checklist scores was demonstrated for TFR and carpal tunnel release; between year in training and global rating scores for TFR and distal radius fracture fixation; and between year in training and pass/fail assessment for TFR. Criterion validity was demonstrated by the correlation between global rating scale scores, detailed checklist scores, and pass/fail assessment for TFR, carpal tunnel release, and distal radius fracture fixation. Time to complete the surgery was not correlated with surgical performance. Residents rated the multiple-station Objective Structured Assessment of Technical Skills format as highly educational. This study reports that a surgeon's ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a resident's abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components. Therapeutic II.

Inter-Rater Reliability: Comparison of Checklist and Global Scoring for OSCEs

Objective Structured Clinical Examinations (OSCEs) have been used globally in evaluating clinical competence in the education of health professionals. Despite the objective intent of OSCEs, scoring methods used by examiners have been a potential source of measurement error affecting the precision with which test scores are determined. In this study, we investigated the differences in the inter-rater reliabilities of objective checklist and subjective global rating scores of examiners (who were exposed to an online training program to standardise scoring techniques) across two medical schools. Examiners’ perceptions of the e-scoring program were also investigated. Two Australian universities shared three OSCE stations in their end-of-year undergraduate medical OSCEs. The scenarios were video-taped and used for on-line examiner training prior to actual exams. Examiner ratings of performance at both sites were analysed using generalisability theory. A single facet, all random persons by raters design [PxR] was used to measure inter-rater reliability for each station, separate for checklist scores and global ratings. The resulting variance components were pooled across stations and examination sites. Decision studies were used to measure reliability estimates. There was no significant mean score difference between examination sites. Variation in examinee ability accounted for 68.3% of the total variance in checklist scores and 90.2% in global ratings. Rater contribution was 1.4% & 0% of the total variance in checklist score and global rating respectively, reflecting high inter-rater reliability of the scores provided by co-examiners across the two schools. Score variance due to interaction and residual error was larger for checklist scores (30.3% vs 9.7%) than for global ratings. Reproducibility coefficients for global ratings were higher than for checklist scores. Survey results showed that the e-scoring package facilitated consensus on scoring techniques. This approach to examiner training also allowed examiners to calibrate the OSCEs in their own time. This study revealed that inter-rater reliability was higher for global ratings than for checklist scores, thus providing further evidence for the reliability of subjective examiner ratings.

The Anatomy of Meckel's Cave: A New Landmark for Innovative Surgical Approaches

The American Board of Surgery encourages graduating medical students to prepare for surgical residency before day 1. We sought to determine the impact of personalized video feedback on an advance preparation task.We conducted a nonrandomized study comparing video feedback versus no feedback. We sent incoming surgical interns a preparatory package 2 months before starting residency. Trainees video-recorded themselves performing a subcuticular wound closure, 3 times at 3-week intervals, and submitted these for appraisal. A staff surgeon provided personalized feedback on each video as a narrated voiceover. The voiced-over videos were then returned to trainees. We compared performance (time and completion rate) on suturing in a multistation assessment against residents from the previous year (no-feedback group).The feedback group had a higher completion rate for the suturing assessment than the no-feedback group (23/28 [82%] vs. 8/27 [30%], P < .0001). The feedback group also completed the suturing station at a faster rate than those without feedback (hazard ratio 4.9 [95% confidence interval (CI): 2.2,11.2), P < .0001). Global rating scores were significantly higher for the feedback group (mean difference [5-point scale] = 0.7 [95% CI: 0.3, 1.1]). However, Objective Structured Assessment of Technical Skills scores indicated no significant difference between groups (mean difference [5-point scale] = 0.3 [95% CI: 0.0, 0.6]). Within the feedback group, we found significant improvement from baseline to final performances (mean difference = 109 seconds [95% CI: 79, 140]).Personalized narrated feedback as part of a home-based advance preparation package for incoming residents is associated with higher performance on early objective assessments.

E-learning interventions are comparable to user's manual in a randomized trial of training strategies for the AGREE II

BackgroundPractice guidelines (PGs) are systematically developed statements intended to assist in patient and practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the AGREE II.MethodsParticipants were randomized to one of three training conditions. 'Tutorial'--participants proceeded through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. 'Tutorial + Practice Exercise'--in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, participants received feedback on how their scores compared to expert norms and formative feedback if scores fell outside the predefined range. 'AGREE II User's Manual PDF (control condition)'--participants reviewed a PDF copy of the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II.ResultsNo differences emerged between training conditions on any of the outcome measures.ConclusionsWe believe these results can be explained by better than anticipated performance of the AGREE II PDF materials (control condition) or the participants' level of health methodology and PG experience rather than the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new to this field; however, this requires further study.

Cognitive and clinical outcomes of homocysteine‐lowering B‐vitamin treatment in mild cognitive impairment: a randomized controlled trial

Homocysteine is a risk factor for Alzheimer's disease. In the first report on the VITACOG trial, we showed that homocysteine-lowering treatment with B vitamins slows the rate of brain atrophy in mild cognitive impairment (MCI). Here we report the effect of B vitamins on cognitive and clinical decline (secondary outcomes) in the same study. This was a double-blind, single-centre study, which included participants with MCI, aged ≥ 70 y, randomly assigned to receive a daily dose of 0.8 mg folic acid, 0.5 mg vitamin B(12) and 20 mg vitamin B(6) (133 participants) or placebo (133 participants) for 2 y. Changes in cognitive or clinical function were analysed by generalized linear models or mixed-effects models. The mean plasma total homocysteine was 30% lower in those treated with B vitamins relative to placebo. B vitamins stabilized executive function (CLOX) relative to placebo (P = 0.015). There was significant benefit of B-vitamin treatment among participants with baseline homocysteine above the median (11.3 µmol/L) in global cognition (Mini Mental State Examination, P < 0.001), episodic memory (Hopkins Verbal Learning Test-delayed recall, P = 0.001) and semantic memory (category fluency, P = 0.037). Clinical benefit occurred in the B-vitamin group for those in the upper quartile of homocysteine at baseline in global clinical dementia rating score (P = 0.02) and IQCODE score (P = 0.01). In this small intervention trial, B vitamins appear to slow cognitive and clinical decline in people with MCI, in particular in those with elevated homocysteine. Further trials are needed to see if this treatment will slow or prevent conversion from MCI to dementia.

Rescuing A Patient In Deteriorating Situations (RAPIDS): An evaluation tool for assessing simulation performance on clinical deterioration

Aim This article is a report of a study which developed and tested the validity and reliability of the RAPIDS-Tool to measure student nurses’ simulation performance in assessing, managing and reporting of clinical deterioration. Background The importance for nurses to recognize and respond to deteriorating patients has led educators to advocate for increasing use of simulation for developing this competency. However, there is a lack of evaluation tools to objectively evaluate nurses’ simulation performance on clinical deterioration. Method The study was conducted in three phases. Phase 1 began with development of items for the RAPIDS-Tool from the basis of a literature review and a panel of national experts’ consensus. Phase 2 established the content validity of the RAPIDS-Tool by a panel of international experts and by undertaking a pilot test. Phase 3 involved testing the psychometric properties of the RAPIDS-Tool, on 30 video-recorded simulation performances, for construct validity, inter-rater reliability, and correlation between two scoring systems. Results The process of development and validation produced a 42-item RAPIDS-Tool. Significant differences ( t = 15.48, p < 0.001) in performance scores among participants with different levels of training supported the construct validity. The RAPIDS-Tool demonstrated a high inter-rater reliability (ICC = 0.99) among the three raters and a high correlation between the global rating and checklist scores ( r = 0.94, p < 0.001). Conclusion The RAPIDS-Tool provides a valid and reliable tool to evaluate nurses’ simulation performances in clinical deterioration. This may prove useful for future studies that investigate outcomes of simulation training.