Competency-based Evaluation Tool Research Articles

749 Medical Student Milestones for Neurosurgery Sub-Interns: A Pilot Study of a Novel Competency-Based Evaluation Tool to Quantitatively Differentiate Residency Applicants

INTRODUCTION: Resident selection is guided by few objective, quantitative datapoints. Utility of a standardized letter of recommendation (SLOR) has been limited by “grade inflation”. The Milestones program in neurosurgery residency standardizes “competence” definitions, recognizes domains of competence beyond patient care and medical knowledge, and highlights deficient areas. A similar approach for neurosurgery sub-interns could provide objective performance data to aid applicant evaluation. METHODS: Medical student milestones were adapted from the residency milestones or newly created according to group consensus of relevant competencies. Competency levels were graded ordinally 1 (level of 3rd-year medical student) to 4 (2nd-year resident). Faculty and self-evaluations were completed for 35 sub-interns across eight programs. Evaluators were surveyed on its usefulness. Average and cumulative milestone scores (CMS) within and across programs were evaluated using descriptive statistics. Kendall’s W tested inter-rater agreement. CMS were compared against percentile assignments on LOR using analysis of variance with post-hoc testing. RESULTS: The average faculty rating overall was 3.20, similar to that expected of an intern. Students scored highest in coachability and feedback (3.49; 3.67) and lowest in bedside procedural aptitude (2.90; 2.85) in faculty and self-evaluations. Median CMS was 26.5 (IQR 21.75-29.75; range 14-32). CMS in four programs evaluating at least five students ranged at least 11 points. A program with three faculty raters demonstrated scoring agreement across five students (p = 0.024). CMS differed significantly between SLOR percentile assignments, despite 25% of students assigned to top 5%ile while CMS-driven assignment of percentiles significantly differentiated quartiles of students. Faculty and students strongly endorsed the milestones form. CONCLUSIONS: The medical student milestones form was well-received and considered useful in objectively differentiating students within and across programs.

Using Simulation to Assess Nurse Practitioner Education.

Resource restraints and the movement toward competency-based education encourage exploration of innovative simulation experiences in advanced practice registered nurse education. Building upon existing best practices in simulation development, this pilot project explores opportunities and challenges translating these practices to the advanced practice registered nurse context. Innovations focused on using a competency-based framework, developing a competency-based evaluation tool, and providing feedback from faculty and standardized patients. Lessons learned from this experience inform recommendations for how to design instructional experiences and provide formative methods of feedback for nurse practitioner student assessment.

An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise

An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise

An Electronic Competency-Based Evaluation Tool for Assessing Humanitarian Competencies in a Simulated Exercise.

The evaluation tool was first derived from the formerly Consortium of British Humanitarian Agencies' (CBHA; United Kingdom), now "Start Network's," Core Humanitarian Competency Framework and formatted in an electronic data capture tool that allowed for offline evaluation. During a 3-day humanitarian simulation event, participants in teams of eight to 10 were evaluated individually at multiple injects by trained evaluators. Participants were assessed on five competencies and a global rating scale. Participants evaluated both themselves and their team members using the same tool at the end of the simulation exercise (SimEx). All participants (63) were evaluated. A total of 1,008 individual evaluations were completed. There were 90 (9.0%) missing evaluations. All 63 participants also evaluated themselves and each of their teammates using the same tool. Self-evaluation scores were significantly lower than peer-evaluations, which were significantly lower than evaluators' assessments. Participants with a medical degree, and those with humanitarian work experience of one month or more, scored significantly higher on all competencies assessed by evaluators compared to other participants. Participants with prior humanitarian experience scored higher on competencies regarding operating safely and working effectively as a team member. This study presents a novel electronic evaluation tool to assess individual performance in five of six globally recognized humanitarian competency domains in a 3-day humanitarian SimEx. The evaluation tool provides a standardized approach to the assessment of humanitarian competencies that cannot be evaluated through knowledge-based testing in a classroom setting. When combined with testing knowledge-based competencies, this presents an approach to a comprehensive competency-based assessment that provides an objective measurement of competency with respect to the competencies listed in the Framework. There is an opportunity to advance the use of this tool in future humanitarian training exercises and potentially in real time, in the field. This could impact the efficiency and effectiveness of humanitarian operations. Evans AB , Hulme JM , Nugus P , Cranmer HH , Coutu M , Johnson K . An electronic competency-based evaluation tool for assessing humanitarian competencies in a simulated exercise. Prehosp Disaster Med. 2017;32(3):253-260.

Teaching leadership in trauma resuscitation: Immediate feedback from a real-time, competency-based evaluation tool shows long-term improvement in resident performance.

Limited data exist on how to develop resident leadership and communication skills during actual trauma resuscitations. An evaluation tool was developed to grade senior resident performance as the team leader during full-trauma-team activations. Thirty actions that demonstrated the Accreditation Council for Graduate Medical Education core competencies were graded on a Likert scale of 1 (poor) to 5 (exceptional). These actions were grouped by their respective core competencies on 5 × 7-inch index cards. In Phase 1, baseline performance scores were obtained. In Phase 2, trauma-focused communication in-services were conducted early in the academic year, and immediate, personalized feedback sessions were performed after resuscitations based on the evaluation tool. In Phase 3, residents received only evaluation-based feedback following resuscitations. In Phase 1 (October 2009 to April 2010), 27 evaluations were performed on 10 residents. In Phase 2 (April 2010 to October 2010), 28 evaluations were performed on nine residents. In Phase 3 (October 2010 to January 2012), 44 evaluations were performed on 13 residents. Total scores improved significantly between Phases 1 and 2 (p = 0.003) and remained elevated throughout Phase 3. When analyzing performance by competency, significant improvement between Phases 1 and 2 (p < 0.05) was seen in all competencies (patient care, knowledge, system-based practice, practice-based learning) with the exception of "communication and professionalism" (p = 0.56). Statistically similar scores were observed between Phases 2 and 3 in all competencies with the exception of "medical knowledge," which showed ongoing significant improvement (p = 0.003). Directed resident feedback sessions utilizing data from a real-time, competency-based evaluation tool have allowed us to improve our residents' abilities to lead trauma resuscitations over a 30-month period. Given pressures to maximize clinical educational opportunities among work-hour constraints, such a model may help decrease the need for costly simulation-based training. Therapeutic study, level III.

A Reliable and Valid Objective Structured Assessment of Technical Skill for the Application of a Pavlik Harness Based on International Expert Consensus.

The use of competency-based training is increasing in medical education as it offers individualized learning opportunities to master required skills. Inherent to this method of teaching is the need for standardized and objective assessments of skill mastery. In orthopaedic surgery, educational programs have focused on surgical skills with lesser emphasis on nonoperative techniques. Accordingly, formal evaluation tools specific to Pavlik Harness application do not exist, despite its widespread use and potential complications from inappropriate application. This study sought to develop a reliable and valid evaluation tool based on international expert consensus to standardize and evaluate Pavlik Harness application for developmental dysplasia of the hip. Consensus was sought from 10 content experts using Delphi methodology. Resulting items formed the Objective Structured Assessment of Technical Skill (OSATS). Thirty-five participants were selected into 3 a priori groups (expert, intermediate and novice) based on Pavlik Harness experience. On 2 occasions, 3 content experts assessed randomized and deidentified videotapes of each participant applying a Pavlik Harness to an infant model using the OSATS and global rating scales (GRS). The reliability and validity of the OSATS was determined with intraclass (ICC) and Pearson correlations and analysis of variance (ANOVA). Consensus was obtained after 2 rounds of structured surveying and resulted in a 25-item OSATS. The reliability of the OSATS was excellent with an ICC of 0.96 for interrater and 0.98 for test-retest reliability. Construct validity was excellent with high correlations between OSATS and GRS (>0.90). In addition, the OSATS discriminated between expert, intermediate, and novice users. We have developed a competency-based evaluation tool for Pavlik Harness application based on consensus from international experts. The OSATS has been shown to be a reliable and valid method for assessing Pavlik Harness application that can discriminate between expert, intermediate, and novice users. Level II.

Calibrating field practicum assessment in social work education with a competency-based evaluation tool in Hong Kong

As social work is an international profession, it is necessary to establish the validity of assessment of the field practicum of students for the purpose of professional accreditation. This study calibrates an indigenous assessment tool, the Social Work Practicum Assessment (SWPA), developed in Hong Kong with a competency-based evaluation (CBE) tool popularized in North America, using data collected from 171 social work final-year undergraduates. The results demonstrate convergence between the SWPA and CBE when rated by field instructors and were greater when the student-assessed CBE was higher. Alternatively, the student-assessed CBE displayed greater convergence with the instructor-rated CBE when the indigenous assessment was higher. The positive results imply the generalizability of the assessments across places.

The State of the Service: A Survey of Psychiatry Resident Education in Psychosomatic Medicine

Although required by the Accreditation Council for Graduate Medical Education, training of general psychiatry residents in Psychosomatic Medicine (PM) varies significantly between programs. In 1996, the Academy of Psychosomatic Medicine (APM) developed guidelines for residency training in PM. Since then, there has been no assessment of the status of PM training during psychiatry residency. Assessment of the current state of PM training in U.S. psychiatry residency programs. A 46-item questionnaire was sent via e-mail to 206 residency directors. Four major areas were assessed: the timing and duration of the PM rotation, level of faculty supervision, didactic curriculum, and role (or potential role) of the APM in residency education. Ninety-two surveys were returned (response rate 45%). Forty-four (54%) of the general psychiatry residencies reported the total duration of the C-L rotation as being between 3 and 6 months (including both full- and part-time rotation). Only 38 (46%) programs' residents complete their PM experience in 1 year of residency. The average Full-Time Equivalent of teaching faculty per service was 1.74 (standard deviation 0.92). Sixty-four (77%) programs have a formal didactic curriculum in C-L. Eighty-one (98%) respondents were aware of the APM. Fifty-eight (70%) had APM members among faculty. The most popular responses on how the APM could best serve training programs were the creation of a subspecialty curriculum (73%) as well as the development of a competency-based evaluation tool (66%). There is significant variation in how residents are taught PM during their training. The APM is a well-recognized organization that may define what constitutes adequate residency training in PM and may help programs fulfill the educational needs of residents.