Standardized Competency Assessment Research Articles

Critical Care Ultrasound Competency of Fellows and Faculty in Pulmonary and Critical Care Medicine: A Nationwide Survey

Purpose: Competency assessment standards for Critical Care Ultrasonography (CCUS) for Graduate Medical Education (GME) trainees in pulmonary/critical care medicine (PCCM) fellowship programs are lacking. We sought to answer the following research questions: How are PCCM fellows and teaching faculty assessed for CCUS competency? Which CCUS teaching methods are perceived as most effective by program directors (PDs) and fellows. Methods: Cross-sectional, nationwide, electronic survey of PCCM PDs and fellows in accredited GME training programs. Results: PDs and fellows both reported the highest rates of fellow competence to use CCUS for invasive procedural guidance, but lower rates for assessment of deep vein thrombosis and abdominal organs. 54% and 90% of PDs reported never assessing fellows or teaching faculty for CCUS competency, respectively. PDs and fellows perceived hands-on workshops and directly supervised CCUS exams as more effective learning methods than unsupervised CCUS archival with subsequent review and self-directed learning. Conclusions: There is substantial variation in CCUS competency assessment among PCCM fellows and teaching faculty nationwide. The majority of training programs do not formally assess fellows or teaching faculty for CCUS competence. Guidelines are needed to formulate standardized competency assessment tools for PCCM fellowship programs.

Competency Assessment of Pediatric Intensive Care Nurses

One of the biggest challenges that clinical nurse educators at the unit level face today is measuring the ongoing competency of nursing staff members to ensure provision of high-quality, evidence-based care. Pediatric nursing leaders at an urban, level I trauma teaching institution in the southwestern United States used a shared governance approach to create a standardized competency assessment tool for pediatric intensive care unit nurses. Donna Wright’s competency assessment model was used as a framework to guide the tool’s development. The adoption of the standardized competency assessment tool was aligned with the organization’s institutional goals and enabled clinical nurse educators to comprehensively evaluate staff members on a regular basis. This standardized competency assessment system for pediatric intensive care nurses is more effective than use of a practice-based, task-oriented assessment method and has improved nursing leaders’ ability to staff the pediatric intensive care unit safely.

Psychometric properties of the Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP) in a Turkish sample.

Cognitive behavioural therapy (CBT) training is highly demanded by clinicians; however, a standardized competence assessment for professionals working with children and young people (CYP) can be problematic. Psychometric tools used for this purpose are typically adult oriented measures. The present study provides psychometric properties of the Cognitive Behaviour Therapy Scale for Children and Young People (CBTS-CYP) derived from a comparative analysis with the Cognitive Therapy Scale (CTS). The CBTS-CYP was used alongside CTS in a CBT supervision training course, consisting of a cohort of 51 therapists. A total of 36 audio/video recorded full CBT sessions were assessed for the purpose of determining CBT competence and adherence to the theory and model. The training involved a total of 80 hours of supervision in 10 meetings via an online videoconferencing platform between July 2020 and February 2021. Face validity and inter-rater reliability of CBTS-CYP were high, with the intraclass correlation values being good (0.60-0.74) or excellent (0.74 and above); the correlations of each CBTS-CYP and CTS items were significant; internal consistency of the scale showed that Cronbach alpha values for total-scale and its two subdimensions were above .93. For a cut-off score of 55 out of 90, sensitivity reached 90.73% and specificity 90.73%. A score of 2 points or above from any single item could be considered as the second minimum criterion for competence. CBTS-CYP offers a valid and reliable scale to evaluate the competence and adherence quality of CBT sessions with children and adolescents.

Strategies to Implement a Competency Assessment Verification Program.

Professional role competence is an essential element of nursing practice and an integral component of providing safe perioperative patient care. In the health care setting, verifying professional role competence and managing the associated documentation can be complex. Educators can use a variety of modalities (eg, flipped classrooms, gaming, podcasts) to present information in a manner that supports adult learning principles. When developing a competency assessment verification program, perioperative leaders should use a structured model to provide consistency; they also should partner with staff members and other key stakeholders (eg, surgeons, risk management personnel) to identify and prioritize ongoing competencies. The leaders and educators should identify competency verification methods, and leaders should designate qualified observers if needed. Documentation of competency activities should be stored in an easily accessible location. Implementing a standardized competency assessment verification program is a best practice that should result in improved patient outcomes.

Pediatric flexible airway endoscopy training during a pandemic and beyond: Bending the curve.

Flexible airway endoscopy, for diagnostic and therapeutic purposes, is a vital aspect of pediatric pulmonology practice, and has become an integral aspect of formal training in pediatric pulmonology. This article is protected by copyright. All rights reserved.

Impact of trainee involvement on the outcome of ERCP procedures: results of a prospective multicenter observational trial.

Training in advanced endoscopic techniques such as endoscopic retrograde cholangiopancreatography (ERCP) should be driven by key performance measures and standardized competence assessment in order to provide safe and high-quality interventions. We aimed to determine whether the involvement of trainees influences the outcome of the procedure and the incidence of ERCP-related adverse events. This was an international, multicenter, prospective, observational study conducted at six high- and low-volume centers across Europe between October 2016 and October 2018, and included independent operators and their trainees. Standard report forms documenting indication, trainee involvement, technical outcome, and complications over a 30-day follow-up of consecutive ERCP procedures were included in the analysis. Technical success of the procedure and procedure-related adverse events were compared between procedures in the trainee group and the control group using bivariable and multivariable analysis. 21 trainees and 16 control endoscopists performed 1843 ERCPs during the study period. Trainee involvement in ERCP procedures did not decrease technical success (92.4 % vs. 93.7 %; P = 0.30) or increase the risk of adverse events (14.7 % vs. 14.6 %; P > 0.99). Conversely, there were significantly more moderate or severe adverse events in the control group compared with the trainee group (6.2 % vs. 3.4 %, P = 0.01). On multivariable analysis, only increased bilirubin levels, time to cannulation, and procedure difficulty level increased the risk of any procedure-related adverse event. Trainee involvement in ERCP interventions within a proper teaching setting is safe and does not compromise the success of the procedure.

Informed consent instead of assent is appropriate in children from the age of twelve: Policy implications of new findings on children's competence to consent to clinical research.

BackgroundFor many decades, the debate on children’s competence to give informed consent in medical settings concentrated on ethical and legal aspects, with little empirical underpinnings. Recently, data from empirical research became available to advance the discussion. It was shown that children’s competence to consent to clinical research could be accurately assessed by the modified MacArthur Competence Assessment Tool for Clinical Research. Age limits for children to be deemed competent to decide on research participation have been studied: generally children of 11.2 years and above were decision-making competent, while children of 9.6 years and younger were not. Age was pointed out to be the key determining factor in children’s competence. In this article we reflect on policy implications of these findings, considering legal, ethical, developmental and clinical perspectives.DiscussionAlthough assessment of children’s competence has a normative character, ethics, law and clinical practice can benefit from research data. The findings may help to do justice to the capacities children possess and challenges they may face when deciding about treatment and research options. We discuss advantages and drawbacks of standardized competence assessment in children on a case-by-case basis compared to application of a fixed age limit, and conclude that a selective implementation of case-by-case competence assessment in specific populations is preferable. We recommend the implementation of age limits based on empirical evidence. Furthermore, we elaborate on a suitable model for informed consent involving children and parents that would do justice to developmental aspects of children and the specific characteristics of the parent-child dyad.SummaryPrevious research outcomes showed that children’s medical decision-making capacities could be operationalized into a standardized assessment instrument. Recommendations for policies include a dual consent procedure, including both child as well as parents, for children from the age of 12 until they reach majority. For children between 10 and 12 years of age, and in case of children older than 12 years in special research populations of mentally compromised patients, we suggest a case-by-case assessment of children’s competence to consent. Since such a dual consent procedure is fundamentally different from a procedure of parental permission and child assent, and would imply a considerable shift regarding some current legislations, practical implications are elaborated.

The Council of Academic Family Medicine procedural and maternity care training guidelines: a better path to consistency in competency assessment in family medicine.

As an organization devoted to training residents to deliver high-quality family medicine care to their communities, we have struggled to determine those procedures in which we should require all residents to develop competency. To date we have lacked consistency in educational standards for both procedural and maternity care training. This lack of standardization has led to a wide range of skills (or lack thereof) in our graduates, which has impacted our scope of care and potentially endangered our credibility as a specialty. A consistent methodology in determining competency has also been lacking. The latest guidance by the RC-FM is, “Residents must receive training to perform clinical procedures required for their future practices in ambulatory and hospital environments.”1 In the FAQ related to this, the RC-FM states, “As the list of procedures performed by the practicing family physicians varies based upon the needs of the community, the program directors and members of the faculty should develop a list of required procedures based upon the needs of their FMP (family medicine practice) and recommendations of organizations…”2 In response, the Council of Academic Family Medicine (CAFM) formed 2 task forces in the spring of 2014. The AFMRD took the lead on developing these guidelines, working with faculty members across the country to provide input into the process. The Society of Teachers of Family Medicine (STFM) Maternity Care and STFM Hospital and Procedures groups formed much of the task forces. After conference calls, a careful literature review, and collaborative efforts, draft guidelines were completed in December 2014. Next steps: gather broader input from family physician educators, update the guidelines based on this feedback, then return the documents to CAFM for final approval. The task forces agreed upon a better method of determining competency that actually passes the common sense test—blending a minimum experience with a standardized competency assessment tool that breaks down the skills that need to be demonstrated by the trainee. The key feature is not relying on numbers alone and, in fact, the minimum numbers are reserved for the most skilled residents, not for the average resident. Most residents will need to exceed the minimum number for complex procedures before they are ready for competency assessment and to potentially be signed off as ready for independent practice. The Maternity Care Guidelines outline training expectations for the 3 tiers of maternity care already being practiced in our family medicine community. These tiers are designated Ambulatory Maternity Care, Comprehensive Maternity Care, and Advanced Maternity Care. Instead of having a one-size-fits-all requirement from the RC-FM, we will have recommendations that reflect what individual residents are seeking in their training, based on the community in which they intend to practice. However, since maternity care is within the domain of our specialty, all programs are expected to offer Ambulatory Maternity Care training to residents to allow them to possess basic spontaneous delivery skills and sound prenatal care training. Even if a graduate does not plan to provide prenatal care for their patients in their practice, they must still possess knowledge of the medical complications of pregnancy and to be able assess the maternity care their patients may be receiving from another physician. The Comprehensive Maternity Care criteria now include labor management as a key portion of experience requirement. With the current duty hour requirements, many residents manage women in labor for extensive periods of time, often making complex care decisions, but would receive no credit by credentialing bodies for that experience. The guidelines have a similar experience criteria model as the procedure guidelines—the minimum number of deliveries is 40, but in addition, they should manage an additional 40 patients in labor (that they may not deliver) during their training. The Advanced Maternity Care tier outlines the expectations for training residents, and often fellows, to gain operative obstetrical maternity skills and management of higher risk pregnancies. This robust level of training is often needed in rural and underserved areas of our country and will create skilled providers of maternity care that those communities need. We are entering an era of greater accountability to our communities we serve. Having these training guidelines for maternity care and for procedures will help us ensure we are training skilled family physicians with a sufficiently broad scope to provide care of higher quality that meets more the needs of their patients. We encourage broad adoption of these guidelines and tools in order to enhance both the skills and credibility of our graduates. These working guidelines can be found on the AFMRD website, http://www.afmrd.org.

The Students Portfolio as an Alternative Strategy to the Standardized Competencies Assessment Tests

This paper is part of a wider research, which focuses on education competencies assessment (I+D Research Project, ref. SEJ-2007-66967), funded by the Spanish Ministry of Education. What we show in this paper are data from one of the case studies we carry out as part of this project. We have developed a "collaborative research" to understand and analyze the educational processes taking place in the "learnings of second order" ("learning how to learn"). In this sense, we've performed a didactic experimentation with portfolios of student in a 5th grade classroom; the conclusions presented are the result of this particular study. As a final, we conclude that portfolios of student can be an interesting strategy for educational evaluation; as well as it can become an "alternative strategy" to the standardized competency assessment tests, which predominate in the current educational landscape.

Accuracy of the MacArthur competence assessment tool for clinical research (MacCAT-CR) for measuring children's competence to consent to clinical research.

An objective assessment of children's competence to consent to research participation is currently not possible. Age limits for asking children's consent vary considerably between countries, and, to our knowledge, the correlation between competence and children's age has never been systematically investigated. To test a standardized competence assessment instrument for children by modifying the MacArthur Competence Assessment Tool for Clinical Research (MacCAT-CR), to investigate its reliability and validity, and to examine the correlation of its assessment with age and estimate cutoff ages. This prospective study included children and adolescents aged 6 to 18 years in the inpatient and outpatient departments of allergology, gastroenterology, oncology, ophthalmology, and pulmonology from January 1, 2012, through January 1, 2014. Participants were eligible for clinical research studies, including observational studies and randomized clinical trials. Competence judgments by experts aware of the 4 relevant criteria-understanding, appreciation, reasoning, and choice-were used to establish the reference standard. The index test was the MacCAT-CR, which used a semistructured interview format. Interrater reliability, validity, and dimensionality of the MacCAT-CR and estimated cutoff ages for competence. Of 209 eligible patients, we included 161 (mean age, 10.6 years; 47.2% male). Good reproducibility of MacCAT-CR total and subscale scores was observed (intraclass correlation coefficient range, 0.68-0.92). We confirmed unidimensionality of the MacCAT-CR. By the reference standard, we judged 54 children (33.5%) to be incompetent; by the MacCAT-CR, 61 children (37.9%). Criterion-related validity of MacCAT-CR scores was supported by high overall accuracy in correctly classifying children as competent against the reference standard (area under the receiver operating characteristics curve, 0.78). Age was a good predictor of competence on the MacCAT-CR (area under the receiver operating characteristics curve, 0.90). In children younger than 9.6 years, competence was unlikely (sensitivity, 90%); in those older than 11.2 years, competence was probable (specificity, 90%). The optimal cutoff age was 10.4 years (sensitivity, 81%; specificity, 84%). The MacCAT-CR demonstrated strong psychometric properties. In children aged 9.6 to 11.2 years, consent may be justified when competence can be demonstrated in individual cases by the MacCAT-CR. The results contribute to a scientific underpinning of regulations for clinical research directed toward children.

Laparoscopic Radical Prostatectomy Demonstrates Less Morbidity Than Open Radical Prostatectomy: An Analysis of the American College of Surgeons-National Surgical Quality Improvement Program Database with a Focus on Surgical Trainee Involvement

Complication rates of open radical prostatectomies (ORPs) and laparoscopic radical prostatectomies (LRPs) performed by highly experienced surgeons in centers of excellence are well known. Using a standardized, national, risk-adjusted surgical database, we compared 30-day outcomes following ORP and LRP and analyzed how trainee involvement influenced outcomes. The American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) is a risk-adjusted data collection analyzing preoperative risk factors, demographics, and 30-day postoperative outcomes. From 2005 to 2011, we identified 10,669 total prostatectomies. Of these, 2278 were ORP and 8391 were LRP. Data on trainee involvement were available on 63% of cases. Comparison of all 10,669 prostatectomies showed a decreased incidence of overall morbidity, serious morbidity, surgical site infections, mortality, wound disruption, urinary tract infection, bleeding, and sepsis or septic shock (p<0.05) for LRP compared with ORP. Trainee involvement was associated with a higher incidence of bleeding, overall and serious morbidity (p<0.001). This difference is isolated to postgraduate year (PGY) 6-10 trainees performing ORP (p<0.001). Overall and serious morbidity was equivalent between PGY groups 1-10 versus attending without trainee performing LRP and PGY groups 1-5 versus attending without trainee performing ORP. Operative times were shorter for ORP versus LRP by an average of 38 minutes (p<0.05), and in cases involving trainees, operative times decreased with trainee experience for both procedures. The length of stay was shorter for LRP compared with ORP (3.2 vs. 1.8 days, p<0.001). The large sample size, standardized data definitions, and quality control measures of the ACS-NSQIP database allow for in-depth analysis of subtle, but significant differences in outcomes between groups. Trainee involvement in LRP appears safe to patients. However, the increased morbidity in ORP involving trainees may be mitigated by awareness, simulation laboratories, and standardized competency assessment.

Performance of speech-language pathology students in problem-based learning tutorials and in clinical practice

The purpose of the study was to identify if performance of speech-language pathology students in problem-based learning (PBL) tutorials could predict subsequent clinical performance evaluated through (a) a non-standardized, custom clinical evaluation form (HKU form) and (b) a standardized competency assessment for speech pathology developed in Australia (COMPASS®). Students’ scores from PBL tutorial performance were correlated with scores in clinical placement on both the HKU form and the COMPASS. Significant correlations were found between students’ PBL tutorial performance (reflective journals and participation in the tutorial process) and their clinical performance (treatment and interpersonal skills) on the HKU clinical evaluation form. Significant correlations were also found between (a) PBL tutorial performance (participation in the tutorial process) and their clinical performance (all generic and occupational competencies, and the overall score) on the COMPASS, (b) PBL tutorial performance (reading forms) and two occupational competencies on the COMPASS, (c) PBL tutorial performance (reflective journals) and four occupational competencies and the overall score on the COMPASS. The results highlighted the need for validating the assessment for the learning process in PBL tutorials with empirical evidence and the advantage of assessing clinical performance through COMPASS in Hong Kong. Tutors, clinical supervisors and students should be given clear behavioral descriptors for expected performance in PBL tutorials and clinical practice at different year levels.

University of Miami Leonard M. Miller School of Medicine

University of Miami Leonard M. Miller School of Medicine

Computer-based animations and static graphics as medical student aids in learning home safety assessment: a randomized controlled trial

Background: Although animations may intuitively seem more effective than static graphics for teaching, there is no clear-cut evidence for the superiority of simple computer-based animations in medical education.Aims: We investigated whether simple animations are better than static graphics as an aid to medical students in learning home safety assessment, an important part of geriatric curriculum.Methods: We used two versions of an interactive online module, one that depicted common home safety issues in static graphics and the other in animations. We randomized first-year medical students who agreed to participate into two groups. After the module, students completed a cognitive burden scale and a standardized competency assessment test in which they had to identify the salient home safety issues and give recommendations based on the hazards. We also captured time spent on task.Results and conclusions: We found no significant differences between the groups in the cognitive burden level, competency assessment scores, and time spent on task. The much cheaper-to-produce static graphics were equally effective as simple animations in this medical education scenario.