Competency-based Assessment Research Articles

Evaluating psychotherapist competence: Testing the generalizability of clinical competence assessments of graduate trainees.

Health service psychology (HSP) graduate programs are shifting from knowledge- to competency-based assessments of trainees' psychotherapy skills. This study used Generalizability Theory to test the dependability of psychotherapy competence assessments based on video observation of trainees. A 10-item rating form was developed from a collection of forms used by graduate programs (n = 102) in counseling and clinical psychology, and a review of the common factors research literature. This form was then used by 11 licensed psychologists to rate eight graduate trainees while viewing 129, approximately 5-min video clips from their psychotherapy sessions with clients (n = 22) at a graduate program's training clinic. Generalizability analyses were used to forecast how the number of raters and clients, and length of observation time impact the dependability of ratings in various rating designs. Raters were the primary source of error variance in ratings, with rater main effects (leniency bias) and dyadic effects (rater-target interactions) contributing 24% and 7% of variance, respectively. Variance due to segments (video clips) was also substantial, suggesting that therapist performance varies within the same counseling session. Generalizability coefficients (G) were highest for crossed rating designs and reached maximum levels (G > .50) after four raters watched each therapist working with three clients and observed 15 min per dyad. These findings suggest that expert raters show consensus in ratings even without rater training and only limited direct observation. Future research should investigate the validity of competence ratings as predictors of outcome. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Physical Therapist Residency Competency-Based Education: Development of an Assessment Instrument.

The objective of this study was to develop generic domains of competence (DoC) with associated competencies and milestones for physical therapist residency education. This work was intended to culminate in establishing validity evidence to support a competency-based assessment instrument that could be used by residency programs to evaluate resident achievement of the competencies, regardless of specialty area. Employing the modified Delphi method, a residency education work group developed an evaluation instrument that included 7 DoC and 31 associated competencies with 6 milestones as rating scales for each competency. The instrument was distributed to mentors and residents in accredited physical therapist residency programs to establish validity evidence. Evaluations (measured by milestones) and demographics were collected at 3 time points (program entry, midterm, and final). Scores across these time points were compared using Kruskal-Wallis tests. Reliability was assessed with kappa statistics (interrater reliability) and alpha reliability coefficients (internal consistency). Construct validity was examined using confirmatory factor analysis via structural equation modeling. Overall, 237 mentors and 228 residents completed 824 evaluations (460 by mentors and 364 resident self-evaluations) across the time points. Scores significantly increased from entry through final time points. The interrater reliability of the associated behaviors ranged from moderate to substantial agreement (κ = 0.417-0.774). The internal consistency was high for all DoC at every time point (α reliability coefficients = .881-.955 for entry, .857-.925 for midterm, and .824-.902 for final). After confirmatory factor analysis with structural equation modeling was performed, a model that included 7 DoC and 20 associated competencies was proposed. The residency assessment instrument developed demonstrates interrater reliability and validity evidence and therefore supports competency-based assessment of resident clinical performance across specialty areas. Additionally, the instrument aligns the physical therapy profession with other professions, such as medicine, dentistry, and pharmacy, that have transitioned to competency-based education. This study outlines the benefits of moving to competency-based education for physical therapist residents, using a sound evaluation tool that evaluates residents across specialty areas. The instrument will allow for transition to competency-based education in physical therapist residency education programs.

The Accreditation Council for Graduate Medical Education Milestones in Integrated Plastic Surgery Programs: How Competency-Based Assessment Has Been Implemented.

The plastic surgery milestones are a central component of resident assessment. The authors performed a survey to evaluate how milestones have been implemented across integrated plastic surgery programs and how faculty perceive the Milestones Project has impacted their program. A 25-question survey was directed to the head of the clinical competency committee of all 82 integrated plastic surgery programs through the American Council of Academic Plastic Surgeons. The survey queried the composition of the committee, how ratings are generated, and the data used in generating these ratings. Committee leaders from 57 programs (69.5 percent) responded to the survey. For most programs (73 percent), one faculty member completed milestone ratings for each resident and reviewed them with the residents. To determine milestone ratings, 92 percent utilized resident assessment after every rotation, and 63 percent utilized in-service examination scores. For documented resident assessment overall, 96 percent of programs assess residents after every rotation; 37 percent asses after every procedure. Feedback is most frequently provided to residents after every rotation (52 percent), rather than only during committee reviews (32 percent) or after every documented assessment (16 percent). Sixty-four percent of respondents did not believe that milestones have helped in the mentorship role. Implementation of the milestones has varied among integrated plastic surgery programs. Structured training for core faculty directed to methods of assessment will make milestones a more effective a tool by which to improve resident evaluation and education. These findings provide a key data set by which to revise the milestones for their second iteration.

Discussion: The Accreditation Council for Graduate Medical Education Milestones in Integrated Plastic Surgery Programs: How Competency-Based Assessment Has Been Implemented.

Discussion: The Accreditation Council for Graduate Medical Education Milestones in Integrated Plastic Surgery Programs: How Competency-Based Assessment Has Been Implemented.

Expert Consensus Guidelines for Assessing Students on the Social Determinants of Health

Phenomenon Assessment and evaluation guidelines inform programmatic changes necessary for educational effectiveness. Presently, no widely accepted guidelines exist for educators to assess learners and evaluate programs regarding social determinants of health (SDOH) during physician and physician assistant (PA) education. We sought to garner expert consensus about effective SDOH learner assessment and program evaluation, so as to make recommendations for best practices related to SDOH education. Approach We used a Delphi approach to conduct our study (September 2019 to December 2020). To administer our Delphi survey, we followed a three-step process: 1) literature review, 2) focus groups and semi-structured interviews, 3) question development and refinement. The final survey contained 72 items that addressed SDOH content areas, assessment methods, assessors, assessment integration, and program evaluation. Survey participants included 14 SDOH experts at US medical schools and PA programs. The survey was circulated for three rounds seeking consensus, and when respondents reached consensus on a particular question, that question was removed from subsequent rounds. Findings The geographically diverse sample of experts reached consensus on many aspects of SDOH assessment and evaluation. The experts selected three important areas to assess learners’ knowledge, skills, and attitudes about SDOH. They identified assessment methods that were “essential”, “useful, but not essential”, and “not necessary.” The essential assessment methods are performance rating scales for knowledge and attitudes and skill-based assessments. They favored faculty and patients as assessors, as well as learner self-assessment, over assessments conducted by other health professionals. Questions about separation versus incorporation of SDOH assessment with other educational assessment did not yield consensus opinion. The experts reached consensus on priority outcome measures to evaluate a school’s SDOH program which included student attitudes toward SDOH, Competence-Based Assessment Scales, and the percentage of graduates involved in health equity initiatives. Insights Based on the Delphi survey results, we make five recommendations that medical and PA educators can apply now when designing learner assessments and evaluating SDOH programming. These recommendations include what should be assessed, using what methods, who should do the assessments, and how they should be incorporated into the curriculum. This expert consensus should guide future development of an assessment and evaluation toolkit to optimize SDOH education and clinical practice. Supplemental data for this article is available online at https://doi.org/10.1080/10401334.2022.2045490 .

A KSA system for competency-based assessment of clinicians’ professional development in China and quality gap analysis

ABSTRACT Background We aim to create a holistic competency-based assessment system to measure competency evolution over time – one of the first such systems in China. Method Two rounds of self-reported surveys were fielded among the graduates from the Shantou University Medical College: June through December 2017, and May through August 2018. Responses from three cohorts of graduates specializing in clinical medicine – new graduates, resident physicians, and senior physicians – were analyzed. Gaps between respondents’ expected and existing levels of competencies were examined using a modified service quality model, SERVQUAL Results A total of 605 questionnaires were collected in 2017 for the construction of competency indicators and a 5-level proficiency rating scale, and 407 in 2018, for confirmatory factor and competency gap analysis. Reliability coefficients of all competency indicators (36) were greater than 0.9. Three competency domains were identified through exploratory factor analysis: knowledge (K), skills (S), and attitude (A). The confirmatory factor analysis confirmed the fit of the scale (CMIN/DF < 4; CFI > 0.9; IFI > 0.9; RMSEA ≤ 0.08). Within the cohorts of resident and senior physicians, the largest competency gap was seen in the domain of knowledge (K): −1.84 and −1.41, respectively. Among new graduates, the largest gap was found in the domain of skills (S) (−1.92), with the gap in knowledge (−1.91) trailing closely behind. Conclusions A competency-based assessment system is proposed to evaluate clinician’s competency development in three domains: knowledge (K), skills (S), and attitude (A). The system consists of 36 competency indicators, a rating scale of 5 proficiency levels, and a gap analysis to measure competency evolution through 3 key milestones in clinician’s professional career: new graduate, resident physician, and senior physician. The competency gaps identified can provide evidence-based guide to clinicians’ own continuous development as well as future medical curriculum improvements.

The ASK-SEAT: a competency-based assessment scale for students majoring in clinical medicine

BackgroundTo validate a competency-based assessment scale for students majoring in clinical medicine, ASK-SEAT. Students’ competency growth across grade years was also examined for trends and gaps.MethodsQuestionnaires were distributed online from May through August in 2018 to Year-2 to Year-6 students who majored in clinical medicine at the Shantou University Medical College (China). Cronbach alpha values were calculated for reliability of the scale, and exploratory factor analysis employed for structural validity. Predictive validity was explored by correlating Year-4 students’ self-assessed competency ratings with their licensing examination scores (based on Kendall’s tau-b values). All students’ competency development over time was examined using the Mann-Whitney U test.ResultsA total of 760 questionnaires meeting the inclusion criteria were analyzed. The overall Cronbach’s alpha value was 0.964, and the item-total correlations were all greater than 0.520. The overall KMO measure was 0.966 and the KMO measure for each item was greater than 0.930 (P < 0.001). The eigenvalues of the top 3 components extracted were all greater than 1, explaining 55.351, 7.382, and 5.316% of data variance respectively, and 68.048% cumulatively. These components were aligned with the competency dimensions of skills (S), knowledge (K), and attitude (A). Significant and positive correlations (0.135 < Kendall’s tau-b < 0.276, p < 0.05) were found between Year-4 students’ self-rated competency levels and their scores for the licensing examination. Steady competency growth was associated with almost all indicators, with the most pronounced growth in the domain of skills. A lack of steady growth was seen in the indicators of “applying the English language” and “conducting scientific research & innovating”.ConclusionsThe ASK-SEAT, a competency-based assessment scale developed to measure medical students’ competency development shows good reliability and structural validity. For predictive validity, weak-to-moderate correlations are found between Year-4 students’ self-assessment and their performance at the national licensing examination (Year-4 students start their clinical clerkship during the 2nd semester of their 4th year of study). Year-2 to Year-6 students demonstrate steady improvement in the great majority of clinical competency indicators, except in the indicators of “applying the English language” and “conducting scientific research & innovating”.

Competency-based assessment of practice-based experiential learning in undergraduate pharmacy programmes.

Objective:To obtain feedback from experiential leaning (EL) leads about how competency-based assessments could be undertaken by EL facilitators, and to scope existing EL assessment structures in undergraduate Masters in Pharmacy (MPharm) programmes across the United Kingdom (UK).Methods:A cross-sectional survey was conducted utilizing a nine-item on-line survey, consisting of five open-ended and four closed-ended question. All UK universities with MPharm programmes (n=30) were invited to participate in the survey. Variables of interest were perceptions on activities and competencies that could be assessed by EL facilitators. The survey utilised a 5-point Likert-type response ranging from strongly disagree to strongly agree. Other variables of interest were tools/methods that could be used to assess competency, and perceived advantages and disadvantages of the proposed methods, the latter two captured via open-ended questions.Results:Of the 21 universities that responded (Response rate: 70%), 17 were included in the final analysis. Fourteen of the 17 (82.4%) offered the 4-year programme, while 3 (17.6%) offered both the 4-year and 5-year integrated programme. Assessments were mainly undertaken by university staff (59%), with minimal amounts undertaken during EL (39%). There was unanimous agreement (100%) that facilitators could assess students’ communication skills and professionalism during EL. No consensus, however, was achieved with regard to the tool(s) or method(s) to be used to assess student’s competencies. There were 13 responses to the open-ended comments. An advantage noted was that EL facilitator assessment of students would allow for more accurate evaluation of students in the practice setting, while acknowledging barriers such as the burden of time and the lack of consistency in marking. To address this lack of consistency, the majority highlighted the need for facilitator training.Conclusion:Minimal assessments are currently undertaken during EL, with students predominantly assessed on return to the university. No consensus could be achieved with regard to the tool(s) or method(s) to be used to assess students’ competencies, suggesting that perhaps there is no one-size-fits-all, and that the tools and methods used should be informed by the competencies being assessed.

A Cuboid Registers Topic, Activity and Competency Data to Exude Feedforward and Continuous Assessment of Competencies

Evaluating competencies achieved by students within a subject and its different topics is a multivariable and complex task whose outcome should provide actual information on their evolution. A relevant feature when a continuous assessment (CA) rules this evaluation is to track their learning process so that pertinent feedforward may be harnessed to proactively promote improvement when required. As this process is performed via a number of activities, such as lectures, problem solving, and lab practice, different competencies are developed, depending on the recurrence and type of conducted activity. Measuring and registering their achievement is the leitmotif of competency-based assessment. In this paper, we assemble topic, activity and competency data into a 3D matrix array to form what we call a TAC cuboid. This cuboid showcases a detailed account of each student evolution, aiding instructors and students to design and follow, respectively, an individualized curricular strategy within a continuous and aligned assessment methodology, which facilitates each student to adequately modify his/her level of development of each competency. In addition, we compare the TAC cuboids’ usage in grading a mathematics subject versus a traditional CA method as well as when a dynamical continuous assessment approach is considered to measure the achievement of mathematical competencies.

Approach to Developing a Core Competency Framework for Student Nurses in Saudi Arabia: Results from Delphi Technique.

Background: Competence, while firmly established as a primary conceptual framework in nursing education, continues to lack clarity and uniformity across borders and contexts. While a wealth of research has been carried out on the various dimensions of this concept, including the drafting and implementation of frameworks for nursing competence, no unifying international framework has been forthcoming. Indeed, the continued development of more localized approaches, based on geography or specialization, would appear to be the most realistic objective. It is incumbent on nurse educationalists and researchers to build on existing frameworks and develop evidence-based tested methodologies for competence assessment in localized contexts. Currently, there is a dearth of such evidence-based frameworks in the Middle East and in the Kingdom of Saudi Arabia (KSA) in particular. This study aimed to formulate and validate a competence framework for undergraduate nursing students in KSA. Results: Following documentary analysis, framework drafting and a three round Delphi process, a consensus was reached as to elements, comprising six discrete domains, to be included. The identified competence domains provide a framework to guide the implementation of a competence-based assessment and move towards a competency-based curriculum for nursing education in KSA. Conclusions: The study concluded that providing a competency-based model and expanding and standardization of competency concept in different dimensions of nursing profession is a necessity; considering that clarification of the concept of competency, the recognition of its dimensions, characteristics and the factors affecting it help in determining the criteria and standardizing the competency tools.

Competency-Based Assessment Methods in Adult Cardiac Sonography: Students’ Perceptions and Clinical Evaluation

Objective: This research examined the effects of multiple combined competency-based methods on sonography students’ perceptions of adult echocardiography training components. In addition, clinical preceptor evaluation scores were compared with faculty objective structured clinical examination (OSCE) scores. Materials and Methods: A quasi-experimental nonequivalent group research design was used to evaluate students enrolled in an adult cardiac Commission on Accreditation of Allied Health Education Programs (CAAHEP) accredited curriculum. Students’ perceptions pre and post multiple competency-based methods (formative assessment, OSCE, &amp; simulation) intervention were recorded via course evaluations. Questions were analyzed individually using descriptive statistics and the Bonferroni correction. Students’ clinical evaluation and OSCE scores were analyzed using Spearman’s rank correlation. Results: The majority of students’ perceptions pre- and postintervention of multiple competency-based assessments demonstrated significant differences, or they primarily agreed that their echocardiography knowledge or skill set was enhanced. There was weak correlation between students’ clinical competency evaluation scores and OSCE scores—post implementation of multiple competency-based assessments, rs(37) = −.01, P = .93. Conclusion: These results suggest further evaluation of the credentialing process’s clinical assessment to ensure clinical competency.

Patient involvement in assessment of postgraduate medical learners: A scoping review.

Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

AAIM Recommendations to Improve Learner Transitions

xxx.

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

BackgroundWorkplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training. MethodsThe ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences. ResultsWe analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test–retest reliability was high (intraclass correlation coefficient [ICC2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent. ConclusionsThis study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

The Implementation of Competency-Based Training and Assessment (CBTA) Framework in Aviation Manpower Planning

A key component of Human Resources Management (HRM) workforce planning at the present stage involves ensuring that employees with the right skills and competencies match the organizational job description. The modern aviation industry has experienced a radical shift in global competition as a diversified network of low-cost and traditional carriers expands the reach, scope, and frequency of traditional passenger-carrying services. The demand for qualified, highly skilled professionals in flight operations and training is rapidly outpacing supply, pressuring airline recruiters to develop innovative strategies for attracting and communicating with job seekers.The study aims to introduce a methodology for implementing the Competency-Based Training and Assessment (CBTA) framework in aviation manpower planning. Research objectives: a theoretical study of the CBTA framework, analysis of the current state of manpower planning, identification of the pilot performance gaps and the role of quality training, and a proposal for implementing the CBTA framework in aviation manpower planning. The authors analyzed the controlling factors and trends in developing the global CBTA framework in the aviation industry.The features of implementing the CBTA framework were considered in view of pandemic consequences. The results of the consequent analysis indicated that during ab initio training, aviation programs should encourage learning beyond the minimum knowledge requirements, providing avenues for continued learning, and focusing on quality rather than quantity (building hours). Based on the research, many findings have been drawn, concluding that human factors demand is a top priority in the aviation industry affected by the organizational culture, resilience to technological changes, and legislation hiring restrictions.The following analysis further proposes a straightforward synthesis of the Lean Six Sigma (L6σ) strategy and implementation of the CBTA framework instead of the traditional approach in many countries, which limits recruitment and selection options, causing unnecessary organizational pressure. Further recommendations include implementing validated CBTA – EBT / psychological assessments - elements during the recruitment process to streamline and support airlines in becoming an employer of choice in the sector.

Counselor Competence Gender Identity Scale: Measuring Bias, Knowledge, and Skills Among School Counselors

The measures presently available to examine the effectiveness of school counselors to provide counseling services to youth and emerging adults who identify as gender diverse are limited. This mixed-methods research study aimed to validate the psychometric properties of the Counselor Competence Gender Identity Scale (CCGIS) with a sample of school counselors. The CCGIS is a competency-based assessment that measures the effectiveness of counselors who provide services to gender diverse individuals in schools. We also examined school counselors’ levels of gender identity counselor competence in relation to region of practice. Considering requests by many gender diverse individuals to have gender-inclusive curriculum in the schools and the negative impact of the COVID-19 pandemic on gender minority youth, we recruited school counselors in the United States to take the CCGIS so its factor structure and validity could be examined. Exploratory factor analysis retained 25 items that formed four subscales with good internal consistency. Criterion and concurrent validity tests indicated that the CCGIS is psychometrically sound. The CCGIS was able to discriminate between groups of school counselors based on region. The CCGIS is a valid tool that can be self-administered by school counselors to assess their effectiveness with gender diverse students.

Effectiveness of a Flight Simulation Training Visual Aid for Normal and Crosswind Approach and Landing

Flight simulators have a significant contribution to effective and efficient flight training across the globe. Significant literature has suggested the effectiveness of simulation tools in training pilots in different scenarios and is an area for continued development and research. To continue those efforts, this study assessed the effectiveness of visual aid to be used as an instructional aid for flight instruction for crosswind landings. The need for further training in crosswind landing is substantiated due to the significantly high number of accidents for General Aviation aircraft in the approach and landing phase of flights. The study utilized an experimental research design and consisted of pre-survey, post-survey, and performance assessments. Along with the descriptive data collected from the surveys, the 5-point competency-based performance assessment was used to grade the landing performance in seven aspects of crosswind landings which was used for the hypothesis testing. The post-results survey highlighted that most participants found that the visual aid assisted with identifying visual cues and perceptions that aided the participants during the crosswind landings in the FTD. Additionally, almost all participants found the visual aid relatively simple to understand and utilize, with all the corresponding lines and marks being easily understood. The statistical analysis of the null hypotheses identified that the visual aid allowed the participants to successfully maintain the required airspeed, vertical speed, and height above the runway threshold on the traffic pattern's final leg. The conclusion of the study highlights the need for such visual aids, further potential improvements in such a visual aid, and scope of utilizing visual aids for improving flight education. While the post-survey and performance assessment highlighted the deficiency in the visual aid in fulfilling some of the objectives of the development of the aid, the results should be used as a foundation for further development of visual aids for flight education.

Society for Improving Medical Professional Learning Collaborative: What’s It All About?

The Society for Improving Medical Professional Learning (SIMPL) Collaborative is a non-profit, educational, quality improvement consortium focused on developing tools, curricula, and policies to improve physician training. The goal is to provide educators and learners with convenient, reliable, and valid evaluation tools for frequent, real-time workplace assessment and feedback. The SIMPL Operating Room application provides this platform. It was developed to provide high-quality, time-sensitive, procedural feedback. Its objective is to facilitate intra-rotation corrections. SIMPL is available to all residency programs and has matured to include over 175 residency programs, involving 19 different specialties, 4000 trainees, 5000 attendings, and 354,800 evaluations in 3 countries. At least 52 peer-reviewed manuscripts have used the evolving database. We have performed an expert narrative review of the entire SIMPL literature (primary research studies, reviews, and websites) to discuss the unique lessons learned from this large collaborative experience. SIMPL can be the core of a competency-based operative skills assessment that is incorporated into medical training, assessment, and certification. Higher quality feedback is provided via SIMPL compared to the routine end-of-rotation evaluations with a corrective comment 60% of the time versus 15%, respectively. A high overall correlation between residents and faculty within case complexity has also been documented (r = 0.76, P &lt; .0001), technical performance (r = 0.66, P &lt; .0001), and autonomy (r = 0.56, P &lt; .0001). The goal of the SIMPL initiative is to use resident performance to drive continuous quality improvement of the individual, programs, and larger medical system.

Workplace-based assessment in the era of competency based medical education: Perception and experiences as a medical teacher

Context: Workplace-based assessment (WPBA) is an essential component of competency-based assessment addressing the “does” level. The study was planned to understand the perceptions and experiences of medical faculties with WPBA for its formal adoption under CBME. Aim: To obtain faculty perceptions and their experiences with WPBA as a medical educator. Settings and Design: A cross-sectional survey was conducted as a part of an online web discussion on WPBA among 38 faculties. Materials and Methods: The information was collected through an online semi-structured questionnaire and data were analyzed using percentages, proportions, and themes. Results: The majority were 31–50 years, females and professors. Directly observed procedural skills (DOPS), case-based discussion (CBD), and logbooks were commonly used tools. The challenges in WPBA practice were hesitancy, time constraints, lack of resources, complexity of tools, and pandemic situation. Conclusions: practice of WPBA tools was perceived positively, however a huge gap was noticed between knowledge and practice of WPBA tools.

Meaningful in-training and end-of-training assessment: The need for implementing a continuous workplace-based formative assessment system in our training programs.

Objectives:To analyze the systems and tools involved in assessment of skill procurement and demonstrating workplace skills in postgraduate medical training.Methods:This cross-sectional survey was carried out by enrolling trainee doctors currently working in Medical, Surgical, Dental and Allied specialties of the country by sending a validated and piloted questionnaire through email. Data collection was done from 20th April to 20th May 2021. Data was analysed using SPSS v. 21.0.Results:A total of 351 completed responses were received from 10 major cities of the country. Multiple aspects of entry-into-training, in-training and end-of-training evaluation showed poor correlation with the required training goals. A comparison of assessment for entry-into-supervised training (FCPS-I) versus independent practice (FCPS-II) showed a dismal situation regarding assessing affective skills like leadership, teamwork, coping with pressure and self-awareness. The concept of maintaining portfolios was completely alien to the trainees and the assessment tools used for demonstrating workplace skills were outdated. The lack of a continuous, periodic and balanced assessment (65%); detailed feedback (61.5%); fair exams (59%); variability in scoring system (58%) and professionalism of the examiners (57.5%) were the most frequently selected perceived flaws in the assessment system by the participants.Conclusion:There are multiple lacunae regarding competency-based assessment systems in our training programs and a massive scope for improvement. Assessment systems should be implemented as continuous process of learning, self-reflection, feedback and revalidation throughout the training tenure at regular and multiple points.