Behavioral Anchors Research Articles

Ability of Ophthalmology Residents to Self-Assess Their Performance Through Established Milestones

Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores. Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores. Wilmer Eye Institute, Johns Hopkins Hospital. Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016. Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms. Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.

The next generation core competencies for emergency management.

The Next Generation Core Competencies (NGCC) guide the professional development of future emergency managers. Once familiar roles are evolving as the world grows more interdependent; at the same time, disaster risk factors are intensified by the changing interactions between the social, built, and physical environments. The updated edition of emergency management core competencies is particularly important for refining the trajectory of the emergency management discipline and developing capacities requisite to reducing disaster risk and building resilient communities in the midst of a turbulent, complex, and uncertain future. The NGCC project was a multiphase study conducted by a FEMA-sponsored focus group. Oriented toward future needs, the competencies have been built on the current emergency management competencies, a review of related competencies and global risk trends, a multiphase Delphi study, and wider emergency management community listening sessions. Behavioral anchors and key actions for measurement accompany the new core competencies. The overarching goal of the work is to establish the next generation emergency management core competencies, which are likely to underpin the emergency management workforce of 2030 and beyond. The 13 core competencies fall into three nested categories that are interrelated, but have attributes that build the individual, the practitioner, or relationships.

Level-$k$ Mechanism Design

Non-equilibrium models of choice (e.g. level-k reasoning) have significantly different, sometimes more accurate, predictions in games than does Nash equilibrium. When it comes to the maximal set of functions that are implementable in mechanism design, however, they turn out to have similar implications. Focusing on single-valued rules, we discuss the role and implications of different behavioural anchors (arbitrary level-0 play), and prove a level-k revelation principle. If a function is level-k implementable given any level-0 play, it must obey a slight weakening of standard strict incentive constraints. Further, the same condition is also sufficient for level-k implementability, although the role of specific level-0 anchors is more controversial for the sufficiency argument. Nonetheless, our results provide tight characterizations of level-k implementable functions under a variety of level-0 play, including truthful, uniform, and atomless anchors.

Trainee Satisfaction with Feedback provided using an Entrustment Scale: A Survey of Internal Medicine Residents

This article was migrated. The article was marked as recommended. Background: Numerous scales have been developed to evaluate medical learners, including entrustment scales. Little is known about resident satisfaction with entrustment scales. The objective of this study was to evaluate residents' perceptions of entrustment scales as a method of assessment in comparison to traditional evaluation methods. Methods: Residents (n=102) at the University of Pennsylvania Internal Medicine residency program were asked to participate in a survey in June 2016 regarding perceptions of an entrustment scale, which was incorporated into end-of-rotation resident clinical evaluations in 2014. The survey assessed the assessment utility in various domains, including overall perception of the scale, overall satisfaction, and preference in comparison with other rating scales. Qualitative comments were elicited via free-text to further clarify residents' perceptions. Results: Sixty of the 102 residents (59%) completed the survey. Most residents positively rated the usefulness of questions asked (n=54, 92%) and comments provided (n=48, 98%). Similarly, of those previously evaluated by numerical scales (n=29), numerical scales with behavioral anchors (n=26), and numerical scales with adjectives (n=35), 72%, 88%, and 83% found the entrustment scale to be equivocal or superior, respectively. Qualitative comments supporting the entrustment scale noted improved ease of interpretation, objectivity, and intuitiveness. Conclusions: Residents are satisfied with entrustment scales, noting the entrustment scale to be superior compared to traditional scale alternatives. Residents found entrustment scales provided a more objective assessment, allowed for easier interpretation, and were more intuitive than traditional scales.

The Development and Implementation of New Assessment Tools for the Surgical Clerkship Rotation.

This article was migrated. The article was marked as recommended. Innovation: We developed two new rubrics with explicit behavioural anchors to assess students in the Queen's undergraduate medical education (UGME) surgery clerkship rotation. These rotation rubrics, complemented by a new ambulatory clinic encounter card, improved the quality, consistency, and timeliness of feedback for clerks from faculty preceptors. This innovation was introduced during a comprehensive workplace-based assessment re-design being undertaken in the Department of Surgery as part of the transition to a post-graduate competency-based medical education (CBME) system for post-graduate education (PGME). The core UGME working group, comprised of a faculty surgeon, assessment consultant, and a surgical resident, selected terminology and designed the tool visual structure to be similar to the new post-graduate assessment tools, since most preceptors supervise learners in both programs. This consistency enhanced buy-in from faculty and ensured a smooth transition to the use of the new UGME tools. Development: The new assessment process was developed and piloted in three phases: (1) development of an assessment system based on rubrics with explicit behavioural descriptors as the key assessment tools; (2) implementation of a pilot study to establish the acceptability and feasibility of the use of these rubrics, with iterative revisions based on stakeholder feedback; and (3) development of a validity argument for the use of these assessment tools. The latter is scheduled for 2018. Outcomes: The use of these rotation behaviour-anchored rubrics and corresponding ambulatory clinic encounter card has greatly improved the mid- and final-rotation feedback provided to students on the Surgery Clerkship. The concrete, descriptive information provided by the rubrics allows the course director to provide specific feedback during rotation exit meetings. The course director has the ability to clearly articulate to students the areas where they have met (or exceeded) the expected level of competency, as well as areas which require additional attention.

An SK BARS System: Ongoing Performance Management With Municipal Police

We present a possible solution to two seemingly paradoxical issues: (a) widespread dissatisfaction with performance management and (b) increased demand for accountability in the public sector. The current article draws from our experience with a municipal police division to clarify and extend Smith and Kendall’s Behaviorally Anchored Rating Scales process to an ongoing performance management system. The resulting system holds promise for being less perfunctory and paternalistic than traditional performance management systems, while fostering transparency and accountability.

A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE.

Faculty members need to assess resident performance using the Accreditation Council for Graduate Medical Education Milestones. In this randomized study we used an objective structured clinical examination (OSCE) around the disclosure of an adverse event to determine whether use of a checklist improved the quality of milestone assessments by faculty. In 2013, a total of 20 anesthesiology faculty members from 3 institutions were randomized to 2 groups to assess 5 videos of trainees demonstrating advancing levels of competency on the OSCE. One group used milestones alone, and the other used milestones plus a 13-item checklist with behavioral anchors based on ideal performance. We classified faculty ratings as either correct or incorrect with regard to the competency level demonstrated in each video, and then used logistic regression analysis to assess the effect of checklist use on the odds of correct classification. Thirteen of 20 faculty members rated assessing performance using milestones alone as difficult or very difficult. Checklist use was associated with significantly greater odds of correct classification at entry level (odds ratio [OR] = 9.2, 95% confidence interval [CI] 4.0-21.2) and at junior level (OR = 2.7, 95% CI 1.3-5.7) performance. For performance at other competency levels checklist use did not affect the odds of correct classification. A majority of anesthesiology faculty members reported difficulty with assessing a videotaped OSCE of error disclosure using milestones as primary assessment tools. Use of the checklist assisted in correct assessments at the entry and junior levels.

Improving patient safety through better teamwork: how effective are different methods of simulation debriefing? Protocol for a pragmatic, prospective and randomised study

IntroductionMedical errors have an incidence of 9% and may lead to worse patient outcome. Teamwork training has the capacity to significantly reduce medical errors and therefore improve patient outcome. One...

Feedback and Assessment Tools for Handoffs: A Systematic Review.

Resident handoff communication skills are essential components of medical education training. There are no previous systematic reviews of feedback and evaluation tools for physician handoffs. We performed a systematic review of articles focused on inpatient handoff feedback or assessment tools. The authors conducted a systematic review of English-language literature published from January 1, 2008, to May 13, 2015 on handoff feedback or assessment tools used in undergraduate or graduate medical education. All articles were reviewed by 2 independent abstractors. Included articles were assessed using a quality scoring system. A total of 26 articles with 32 tools met inclusion criteria, including 3 focused on feedback, 8 on assessment, and 15 on both feedback and assessment. All tools were used in an inpatient setting. Feedback and/or assessment improved the content or organization measures of handoff, while process and professionalism measures were less reliably improved. The Handoff Clinical Evaluation Exercise or a similar tool was used most frequently. Of included studies, 23% (6 of 26) were validity evidence studies, and 31% (8 of 26) of articles included a tool with behavioral anchors. A total of 35% (9 of 26) of studies used simulation or standardized patient encounters. A number of feedback and assessment tools for physician handoffs in several specialties have been studied. Limited research has been done on the studied tools. These tools may assist medical educators in assessing trainees' handoff skills.

Implementation of a Needs-Based, Online Feedback Tool for Anesthesia Residents With Subsequent Mapping of the Feedback to the ACGME Milestones.

Optimizing feedback that residents receive from faculty is important for learning. The goals of this study were to (1) conduct focus groups of anesthesia residents to define what constitutes optimal feedback; (2) develop, test, and implement a web-based feedback tool; and (3) then map the contents of the written comments collected on the feedback tool to the Accreditation Council for Graduate Medical Education (ACGME) anesthesiology milestones. All 72 anesthesia residents in the program were invited to participate in 1 of 5 focus groups scheduled over a 2-month period. Thirty-seven (51%) participated in the focus groups and completed a written survey on previous feedback experiences. On the basis of the focus group input, an initial online feedback tool was pilot-tested with 20 residents and 62 feedback sessions, and then a final feedback tool was deployed to the entire residency to facilitate the feedback process. The completed feedback written entries were mapped onto the 25 ACGME anesthesiology milestones. Focus groups revealed 3 major barriers to good feedback: (1) too late such as, for example, at the end of month-long clinical rotations, which was not useful because the feedback was delayed; (2) too general and not specific enough to immediately remedy behavior; and (3) too many in that the large number of evaluations that existed that were unhelpful such as those with unclear behavioral anchors compromised the overall feedback culture. Thirty residents (42% of 72 residents in the program) used the final online feedback tool with 121 feedback sessions with 61 attendings on 15 rotations at 3 hospital sites. The number of feedback tool uses per resident averaged 4.03 (standard deviation 5.08, median 2, range 1-21, 25th-75th % quartile 1-4). Feedback tool uses per faculty averaged 1.98 (standard deviation 3.2, median 1, range 1-25, 25th-75th % quartile 1-2). For the feedback question item "specific learning objective demonstrated well by the resident," this yielded 296 milestone-specific responses. The majority (71.3%) were related to the patient care competency, most commonly the anesthetic plan and conduct (35.8%) and airway management (11.1%) milestones; 10.5% were related to the interpersonal and communication skills competency, most commonly the milestones communication with other professionals (4.4%) or with patients and families (4.4%); and 8.4% were related to the practice-based learning and improvement competency, most commonly self-directed learning (6.1%). For the feedback tool item "specific learning objective that resident may improve," 67.0% were related to patient care, most commonly anesthetic plan and conduct (33.5%) followed by use/interpretation of monitoring and equipment (8.5%) and airway management (8.5%); 10.2% were related to practice-based learning and improvement, most commonly self-directed learning (6.8%); and 9.7% were related to the systems-based practice competency. Resident focus groups recommended that feedback be timely and specific and be structured around a tool. A customized online feedback tool was developed and implemented. Mapping of the free-text feedback comments may assist in assessing milestones. Use of the feedback tool was lower than expected, which may indicate that it is just 1 of many implementation steps required for behavioral and culture change to support a learning environment with frequent and useful feedback.

Does the Attitudes Towards Daily Activities at Romanian Aba’ Therapists are Related with Their Well-Being Level?

Taking care of people with disabilities it is a noble and demanded profession. It comes with serious responsibilities and with personal and in time, with professional satisfactions. The experts that are working currently with persons with special needs, in particular with children with special have to have specific competencies, but also have to have a “sparkle of creativity” in order to create the personalized therapeutic interventional program with achievable goals and related to all the types the resources that are accessible. Furthermore the number of the children with autism spectrum, in the last decade, is increased and it became very clearly the experts’ necessities for constant professional development. In the last five years, there are a lot of studies (that have been done at specialists involved in the recovery process of autistic children), focusing in revealing, the main behavioural anchors that are “damaged” as the results of the specific daily current activities. This paper tries to reveal the implications of the daily stress activities upon the quality of life at Romanian therapists in order to develop new strategies in managing stressful professional situations.

Do Emergency Medicine Residents Prefer Resident-initiated or Attending-initiated Feedback?

Real-time feedback is crucial to improving physician performance. Emerging theory suggests that learner-initiated feedback may be more effective in changing performance than attending-initiated feedback, but little is known about how residents perceive resident- versus attending-initiated feedback. The primary aim was to determine whether residents' satisfaction varied by learner-versus attending-initiated feedback encounters. We hypothesized that residents would be more satisfied with resident-initiated feedback. This was a multicenter study of five emergency medicine residency programs. We developed a milestones-based, real-time feedback intervention that provided behavioral anchors for ED subcompetencies and prompted a feedback discussion. The intervention was implemented at all sites for a 3-month period from March to November 2014. Residents were asked to initiate one card per shift; attendings were also invited to initiate encounters and, in either instance, asked to provide one specific suggestion for improvement. Residents confidentially rated their satisfaction with feedback on a 10-point scale. Reported satisfaction was categorized as "very satisfied" (score of 10) versus "less than very satisfied" (score<10). Logistic regression was used to assess the difference in satisfaction between resident- versus attending-initiated feedback, and random effects were used to account for the clustering of repeated ratings within resident and by site. A total of 785 cards was collected from five sites. Participation varied by site (range= 21-487 cards per site). Of the 587 cards with both feedback initiator and satisfaction data, 67% (396/587) were resident-initiated, and median satisfaction score was 10 (range= 4-10). There was no difference in the odds of being "very satisfied" by resident- vesus attending-initiated encounters (odds ratio= 1.08, 95% confidence interval= 0.41 to 2.83). Our results suggest that residents are likely to be as satisfied with self-initiated feedback as attending-initiated feedback. Further research is needed to determine whether resident-initiated feedback is more likely to be incorporated into practice and result in objective performance improvements.

Tabletop Simulation to Assess Teamwork Competency in OB/GYN Resident Candidates

BACKGROUND: Selecting new residents for an OBGYN training program is a high-stakes undertaking for both candidates and the programs to which they apply. Determining which candidates would work well within their cohort is of paramount importance to the success of a program. In addition, the quality of patient care has been shown to be positively impacted by inter-professional collaboration and well-functioning healthcare teams. METHODS: Mr. Potato Head® tabletop simulation was adapted for use during the UVM OB/GYN Residency Interview Day. Six to 7 candidates were assigned to each team. Candidates were assessed by 2 OBGYN residents who received training on identifying behavioral anchors associated with performance. Results of the team work simulation were included in the holistic review of candidates at the completion of the interview day. RESULTS: Seventy-one percent of candidate respondents reported favorable feedback on a post-interview day survey, describing the activity as a unique experience reflecting the value our program places on teamwork. 30/46 candidates met expectations with 6/30 nearly exceeding expectations. 10/46 candidates were rated as borderline meeting expectations and 6/46 failed to meet baseline expectations. Of the 6 candidates rated as not meeting expectations, 3 (50%) were not ranked in our program. 13/14 interviewers found the Potato Head Simulation to be valuable in their assessment of candidates, with 10/14 reporting a moderate to significant influence. Behavior anchors associated with meeting expectations included good listening skills, treating others with respect, the ability to reflect and be open to feedback. DISCUSSION: Teamwork is a skill set that is difficult to evaluate. Traditional one on one-interview scenarios rarely provide the opportunity for candidates to demonstrate their ability to function within a team. Tabletop Simulation provides a feasible, engaging and well received activity during the interview day to assists programs in evaluating candidate team work skills.

Midclerkship feedback in the surgical clerkship: the “Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills” application utilizing learner self-assessment

BackgroundThe Liaison Committee on Medical Education requires midclerkship formative (low stakes) feedback to students regarding their clinical skills. Student self-assessment is not commonly incorporated into this evaluation. We sought to determine the feasibility of collecting and comparing student self-assessment with that of their preceptors using an iPad application. These student self-ratings and preceptor ratings are jointly created and reviewed as part of a face-to-face midclerkship feedback session. MethodsUsing our iPad application for Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills (“PRIMES”), students answer 6 questions based on their self-assessment of performance at midclerkship. Each skill is rated on a 3-point scale (beginning, competent, and strong) with specific behavioral anchors. The faculty preceptors then complete the same PRIMES form during the face-to-face meeting. The application displays a comparison of the 2 sets of ratings, facilitating a discussion to determine individualized learning objectives for the second half of the clerkship. ResultsA total of 209 student–preceptor pairs completed PRIMES ratings. On average, student–preceptor ratings were in agreement for 38% of the time. Agreement between students and preceptors was highest for Professionalism (70%) and lowest for Procedural Skills (22%). On average, 60% of student–preceptor ratings did not agree. Students rated themselves lower than preceptors 52% of the time, while only 8% of students rated themselves higher than their preceptors' ratings (this difference is significant at the P value <.05 level). ConclusionsThis study demonstrates the value of using the PRIMES framework to incorporate surgery clerkship students' self-assessment into formative face-to-face midclerkship feedback sessions with their preceptors with the goal to improve performance during the second half of the clerkship.

Identifying and prioritizing the effecting factors on implementing policies of Medical Science University of Zabol

Policy making University of Medical Sciences, in a comprehensive look, three stages of formulation, implementation and evaluation. In the full implementation of the policies of Medical Sciences, indices are several factors to identify them in order to streamline the implementation of policies is important. In this paper, according to studies, the perspective of managers, supervisors and experts on the factors affecting the implementation of the policies set and looked University of Medical Sciences. In the present study, type research methods analytical and applied in terms of goal. The population of managers, supervisors and reporters are Zahedan and Zabol University of Medical Sciences. The questionnaire that the overall validity of the content and reliable 0.84 with Cronbach’s Alpha coefficient was estimated. The results of this study six important landmark in the implementation of policies were identified and ratings which include administration and bureaucracy, targets and policy development, enforcement expertise, personality traits and behavioral anchors, consumers and supported by the legal authorities. Keywords: public policy making, vision, medical sciences, health document. JEL Classification: D78, I11

Topical Review: Building Competency: Professional Skills for Pediatric Psychologists in Integrated Primary Care Settings.

OBJECTIVES: In the midst of large-scale changes across our nation's health care system, including the Affordable Care Act and Patient-Centered Medical Home initiatives, integrated primary care models afford important opportunities for those in the field of pediatric psychology. Despite the extensive and growing attention, this subspecialty has received in recent years, a comprehensive set of core professional competencies has not been established. METHODS: A subset of an Integrated Primary Care Special Interest Group used two well-established sets of core competencies in integrated primary care and pediatric psychology as a basis to develop a set of integrated pediatric primary care-specific behavioral anchors. CONCLUSIONS: The current manuscript describes these behavioral anchors and their development in the context of professional training as well as with regard to Triple Aim goals and securing psychology's role in integrated pediatric primary care settings.

Evaluating Interprofessional Team Performance: A Faculty Rater Tool.

Reliable team assessment has become a priority because of growing emphasis on interprofessional education and team-based care. Objective rating scales are needed to evaluate interprofessional student teams and individuals and provide real-time feedback. In response to a need for behavioral rating scales, we modified the McMaster-Ottawa Scale from a 9-point to a 3-point scale and added descriptive behavioral anchors to define three levels of competency (i.e., below, at, and above expected). This modification is intended to provide consistent rating of individuals and teams in patient settings. We then developed a demonstration video using actors representing four professions to demonstrate the three levels of performance within the team. Our faculty rater tool, consisting of the modified scale and video, is designed to provide standardized ratings in interprofessional educational settings that involve patient care. We conducted training sessions with 40 faculty members from seven professions (medicine, dentistry, occupational therapy, nursing, pharmacy, physician assistant, and psychology) over a 2-year period. Immediately after each training session, two trained faculty observers rated interprofessional student teams as they conducted history and assessments on standardized patients. Observer scores were compared with one another and with standard expert ratings of the same teams. Trained observer ratings were consistent across the pairs. The observer training can be conducted within 60-90 minutes with the tool. Results of our implementation of the faculty rater tool confirm that the modified McMaster-Ottawa Scale is feasible to administer in clinical settings and that the demonstration video can be easily adopted for standardizing observer ratings.

Behaviorally anchored rating scales: An application for evaluating teaching practice

We developed Behaviorally Anchored Rating Scales (BARS) for measuring teaching practice, and compared them to the well-established Framework for Teaching (FfT; Danielson, 2013). BARS provide behavioral anchors delineating levels of performance via a set of behaviors. Our BARS focused on two dimensions of teaching, leading a classroom discussion and making content and practices explicit. We examined how a) rater agreement for BARS compares to rater agreement using the FfT, and b) how BARS and the FfT compare regarding perceived ease of use, perceived accuracy, and perceived advantages and disadvantages. Nineteen raters, who are users of the FfT and were trained to use BARS, independently evaluated video-taped teacher lessons using both methods. Rater agreement was higher for the FfT, which may, in part, be a factor of the raters' greater experience with that rating system. Nonetheless, raters reported that there are many aspects of BARS that are desirable.

Validity of ophthalmology surgical competency assessment rubric for strabismus surgery in resident training

Strabismus) tool was designed to aid in the assessment of surgical skills using procedure specific behavioral anchors. The present study evaluated inter-rater agreement of the Strabismus tool in the assessment of resident performance. Strabismus evaluations of resident surgical strabismus cases were performed by a multinational group of faculty strabismus surgeons. Cronbach α statistical analysis of the completed evaluations revealed high inter-rater agreement, indicating the Strabismus is a reliable tool to facilitate assessment of resident strabismus surgical skills.

Continued Validation of the O-SCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the Simulated Environment

Construct: The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) is a 9-item surgical evaluation tool designed to assess technical competence in surgical trainees using behavioral anchors. Background: The initial development of the O-SCORE produced evidence for valid results. Further work is required to determine if the use of a single surgeon or an unblinded rater introduces bias. In addition, the relationship of the O-SCORE to other currently used technical assessment tools should be explored to provide validity evidence related to the relationship to other measures. We have designed this project to provide continued validity evidence for the O-SCORE related to these two issues. Approach: Nineteen residents and 2 staff Orthopedic Surgeons from the University of Ottawa volunteered to participate in a 2-part OSCE style station. Participants completed a written questionnaire followed by a videotaped 10-minute simulated open reduction and internal fixation of a midshaft radius fracture. Videos were rated individually by 2 blinded staff orthopedic surgeons using an Objective Structured Assessment of Technical Skills (OSATS) global rating scale, an OSATS checklist, and the O-SCORE in random order. Results: O-SCORE results appeared sensitive to surgical training level even when raters were blinded. In addition, strong agreement between two independent observers using the O-SCORE suggests that the measure captures a performance easily recognized by surgical observers. Ratings on the O-SCORE also were strongly associated with global ratings on the currently most validated technical evaluation tool (OSATS). Collectively, these results suggest that the O-SCORE generates accurate, reproducible, and meaningful results when used in a randomized and blinded fashion, providing continued validity evidence for using this tool to evaluate surgical trainee competence. Conclusions: The O-SCORE was able to differentiate surgical trainee level using blinded raters providing further evidence of validity for the O-SCORE. There was strong agreement between two independent observers using the O-SCORE. Ratings on the O-SCORE also demonstrated equivalence to scores on the most validated technical evaluation tool (OSATS). These results suggest that the O-SCORE demonstrates accurate and reproducible results when used in a randomized and blinded fashion providing continued validity evidence for this tool in the evaluation of surgical competence in the trainees.