Point-of-care ultrasound training using near-peer training and remote Supervision: A pilot randomized controlled trial.

Background Objective structured clinical examination (OSCE) is used worldwide. This study aims to explore potential alternatives to the OSCE by using entrustable professional activities (EPA)-based assessments in the workplace. Methods This study enrolled 265 six-year undergraduate medical students (UGY) from 2021 to 2023. During their rotations, students were assessed using 13 EPAs, with the grading methods modified to facilitate application. Before graduation, they participated in two mock OSCEs and a National OSCE. We used generalized estimating equations to analyze the associations between the EPA assessments and the OSCE scores, adjusting for age and sex, and developed a prediction model. EPA8 and EPA9, which represent advanced abilities that were not significant in the regression models, were removed from the prediction model. Results Most EPAs were significantly correlated with OSCE scores across the three cohorts. The prediction model for forecasting passing in the three OSCEs demonstrated fair predictive capacity (area under curve = 0.82, 0.66, and 0.71 for students graduated in 2021, 2022, and 2023, respectively all p < 0.05). Conclusions The workplace-based assessments (EPA) showed a high correlation with competency-based assessments in simulated settings (OSCE). EPAs may serve as alternative tools to formal OSCE for medical students.

Little is known regarding the actual utilization of point-of-care ultrasound (PoCUS) after training. This study aims to investigate the clinical utilization of PoCUS by first post-graduate year (PGY-1) residents after training. This prospective study was conducted at the emergency department (ED). A PoCUS curriculum was implemented and the objective structured clinical examination (OSCE) scores were obtained after training. The sonographic examinations performed by the residents were collected. The primary outcomes were the numbers, imaging quality, and accuracy of the sonographic examinations after the curriculum, compared with those before the curriculum. Two hundred and thirtynine residents participated with the median OSCE score of 4 (IQRs, 4-5) and 170 (71%) used PoCUS during clinical practice. The number that each resident performed increased [before vs. after, 0 (0-1) vs. 3 (0-5), p<0.0001], the image quality was better [before vs. after, 3 (2-3) vs. 3 (3), p<0.0001] and the accuracy improved (before vs. after, 117/129 vs. 730/772, p<0.0001) after the curriculum. The residents were categorized into 4 groups based on the utilization: group 1 performed PoCUS before and after the curriculum; group 2 performed only after the curriculum; group 3 performed only before the curriculum; the last did not use. No significant differences existed in the OSCE score between the 4 groups. Group 1 performed more examinationswith better image quality and groups 1 and 2 used ≥2 applications after the curriculum. However, nearly 30% of residents did not use PoCUS, and "chose other imaging priorities" (40/69, 58%) was the main feedback. A PoCUS training had a positive impact on the clinical utilization by the novice residents. More than 70% of residents integrated PoCUS into clinical practice and used self-formatted US techniques. The OSCE scores could not predict further utilization.

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ''problem stations'' were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.

BackgroundThe objective structured clinical examination (OSCE) is a part of emergency medicine (EM) examinations such as the Masters of Medicine in Emergency Medicine (MMed) examination and the equivalent Member of the Royal College of Emergency Medicine (MRCEM) examination. The use of formative OSCEs to prepare EM residents for summative OSCEs has not been investigated. This study aimed to evaluate the role of formative OSCEs in preparing EM residents for the MMed and MRCEM OSCE.MethodsThis was an observational, retrospective, mixed-methods cohort study. We analysed data from formative OSCEs conducted by the National Healthcare Group EM residency programme from 2013 to 2019, and from a questionnaire distributed to all residents during the study period. Residents’ formative OSCE participation and scores were compared with first-attempt summative OSCE success. Qualitative analysis of residents’ opinions on the usefulness of the formative OSCE was performed.ResultsForty-three of the 50 (86.0%) residents attended at least one formative OSCE. Of the 46 who responded to the questionnaire, 40 (87.0%) had attempted and succeeded in the MMed or MRCEM OSCE, of whom 35 (87.5%) had succeeded on the first attempt. Residents who succeeded in the summative OSCE on the first attempt tended to have higher proximate (mean = 70.6, SD = 8.9 vs mean = 64.3, SD = 10.8) and mean (mean = 67.4, SD = 7.1 vs mean = 62.8, SD = 7.3) formative OSCE scores. All 8/40 (20.0%) residents who attended more than three formative OSCEs succeeded in the summative OSCE on their first attempt. Residents’ formative OSCE scores tended to improve with successive formative OSCEs, demonstrating a positive training effect. All residents felt that the formative OSCE was useful in preparing them for the summative OSCE.ConclusionsParticipation in multiple formative OSCEs was beneficial in preparing residents for the summative OSCE. The formative OSCE was useful in familiarising residents with the examination, giving them an opportunity to perform in near-examination conditions, and providing feedback to residents and faculty about their progress. Our findings may support the implementation of formative OSCEs in other training programmes to prepare learners for high-stake summative OSCEs.

To the Editor: We read with great interest the article by Yeates and colleagues 1 addressing the impact of examiner variability on the Objective Structured Clinical Examination (OSCE) scores and using novel technology to combat this. As fourth-year medical students from Imperial College London, this article particularly resonated with us, and we encourage changes to improve the fairness of OSCE scoring. We support the implementation of Video-based Examiner Score Comparison and Adjustment (VESCA) because it not only enhances fairness in OSCE scoring but it also has the potential to revolutionize the structure of the OSCE. As the Internet system is further developed, it could eventually negate the need for a live examiner. During our recent OSCE, we felt that examiners with favorable facial expressions and open body language may have reduced stress and helped students perform better than they would have otherwise. Although management of nonverbal cues may be covered in training, we believe that remote Internet scoring would be a more effective solution. Conversely, VESCA does not take into consideration Simulated Patients’ (SPs) behavior and portrayal of the condition. SPs’ physical and verbal portrayal contributes significantly to how candidates interact with them. Tamblyn and colleagues 2 found that SPs who received 2 training sessions depicted the condition more accurately than those who received 1. We have experienced how SPs can affect candidate scores, as they also grade students during the OSCE. Therefore, SPs must be trained regularly and be assessed during the OSCE. SP regulations, in combination with VESCA, could improve the overall fairness of the OSCE. To further the study and encourage future implementation of VESCA, we suggest adopting an alternative approach to participant recruitment. Voluntary participation for VESCA raises the question of whether students who were more confident in their abilities opted to participate. We suggest that an opt-out approach could be adopted to increase participant numbers and result in a more accurate depiction of the original cohort. In fact, it may be of interest to study if borderline students perform better with unobtrusive cameras rather than a live examiner in the room.

Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education?

Using artificial intelligence for automated assessment of point-of-care ultrasound (POCUS) skills in emergency medicine.

The first aim was to study the reliability of a dental objective structured clinical examination (OSCE) administered over multiple days, and the second was to assess the number of test stations required for a sufficiently reliable decision in three score interpretation perspectives of a dental OSCE administered over multiple days. In four OSCE administrations, 463 students of the year 2005 and 2006 took the summative OSCE after a dental course in comprehensive dentistry. The OSCE had 16-18 5-min stations (scores 1-10), and was administered per OSCE on four different days of 1 week. ANOVA was used to test for examinee performance variation across days. Generalizability theory was used for reliability analyses. Reliability was studied from three interpretation perspectives: for relative (norm) decisions, for absolute (domain) and pass-fail (mastery) decisions. As an indicator of reproducibility of test scores in this dental OSCE, the standard error of measurement (SEM) was used. The benchmark of SEM was set at <0.51. This is corresponding to a 95% confidence interval (CI) of <1 on the original scoring scale that ranged from 1 to 10. The mean weighted total OSCE score was 7.14 on a 10-point scale. With the pass-fail score set at 6.2 for the four OSCE, 90% of the 463 students passed. There was no significant increase in scores over the different days the OSCE was administered. 'Wished' variance owing to students was 6.3%. Variance owing to interaction between student and stations and residual error was 66.3%, more than two times larger than variance owing to stations' difficulty (27.4%). The SEM norm was 0.42 with a CI of +/-0.83 and the SEM domain was 0.50, with a CI of +/-0.98. In order to make reliable relative decisions (SEM <0.51), the use of minimal 12 stations is necessary, and for reliable absolute and pass-fail decisions, the use of minimal 17 stations is necessary in this dental OSCE. It appeared reliable, when testing large numbers of students, to administer the OSCE on different days. In order to make reliable decisions for this dental OSCE, minimum 17 stations are needed. Clearly, wide sampling of stations is at the heart of obtaining reliable scores in OSCE, also in dental education.

Assessment of competence in technical skills, including point-of-care ultrasound (POCUS), is required before a novice can safely perform the skill independently. Ongoing assessment of competence is also required because technical skills degrade over time, especially when they are infrequently performed or complex. Hand-motion analysis (HMA) is an objective assessment tool that has been used to evaluate competency in many technical skills. The purpose of this study was to demonstrate the feasibility and validity of HMA as an assessment tool for competence in both simple and complex technical skills as well as skill degradation over time. This prospective cohort study included 36 paramedics with no POCUS experience and six physicians who were fellowship trained in POCUS. The novices completed a 4-hour didactic and hands-on training program for cardiac and lung POCUS. HMA measurements, objective structured clinical examinations (OSCE), and written examinations were collected for novices immediately before and after training as well as 2 and 4months after training. Expert HMA metrics were also recorded. Expert HMA metrics for cardiac and lung POCUS were significantly better than those of novices. After completion of the training program, the novices improved significantly in all HMA metrics, knowledge test scores, and OSCE scores. Novices showed skill degradation in cardiac POCUS based on HMA metrics and OSCE scores while lung POCUS image acquisition skills were preserved. Novices deemed competent by OSCE score performed significantly better in HMA metrics than those not deemed competent. We have demonstrated that HMA is a feasible and valid tool for assessment of competence in technical skills and can also evaluate skill degradation over time. Skill degradation appears more apparent in complex skills, such as cardiac POCUS. HMA may provide a more efficient and reliable assessment of technical skills, including POCUS, when compared to traditional assessment tools.

Purpose: The purpose of this study is to analyze the development and implementation of the Objective Structured Clinical Examination(OSCE) for the residency applicant examination at Daegu- Catholic Hospital. Methods: Fifty-four Daegu-Catholic residency applicants were evaluated by written examination, internship scores, and OSCE. The correlation between written examination scores, internship scores, and OSCE scores was assessed. Results: The correlation between OSCE and internship scores showed a tendency to be more significant than that between the written test and internship scores, but OSCE-internship correlation coefficients and written test and internship correlation coefficients was not statistically significant. The distribution of OSCE and internship scores on a graph corroborated this relationship between the two variables. Conclusion: The OSCE presents more objective criteria for residency application tests.

The undergraduate MBBS course (Bachelor of Medicine, Bachelor of Surgery) in India is designed to produce competent Indian Medical Graduates (IMGs) capable of addressing community health needs. The preparedness of these graduates for clinical practice remains a crucial area of investigation. Objective Structured Clinical Examinations (OSCEs) have emerged as a robust assessment tool for evaluating clinical competencies. This study aims to assess the feasibility and utility of the OSCE when used as an end-of-internship evaluation, describe temporal trends in OSCE performance, and analyze the correlation between OSCE scores and final-year MBBS examination results over a six-year period at a center in South India. A descriptive study was conducted at a medical college in South India, involving six batches of students (2017-2023; n = 469). OSCE stations were designed to assess cognitive, psychomotor, and affective domains. The feasibility and utility of this exercise was assessed. OSCE scores were analyzed across domains and subdomains, and correlations with final-year MBBS examination scores were explored using Pearson's correlation coefficient. Statistical significance was set at p < 0.05. The OSCE was found to be a resource-intensive evaluation, requiring 528 faculty-hours for a batch of 60 students and 843 faculty-hours for a batch of 150. The mean OSCE score was 61.3 ± 8.0%, with the highest performance in the affective domain (72.1 ± 14.2%) and the lowest in the cognitive domain (50.0 ± 15.4%). Notably, psychomotor skills were inversely correlated with cognitive skills (-0.41, p < 0.001). OSCE scores were weakly correlated with final-year MBBS examination marks (r = 0.44, p < 0.001), with stronger correlations in smaller batches (r = 0.57, p < 0.001) than larger ones (r = 0.44, p < 0.001). The COVID-19 pandemic and increased batch sizes negatively impacted clinical and technical skill performance. It is feasible to conduct OSCEs as an end of internship assessment in a medical college. OSCEs serve as a comprehensive assessment tool, underscoring the importance of multi-modal evaluation strategies. Conducting OSCEs serves as an appropriate assessment of the MBBS training programme so as to subsequently tailor the programme according to performance in various subjects, domains and subdomains.

To assess the educational value of an objective structured clinical examination (OSCE) administered in three consecutive years (1992-93 to 1994-95) to first-year residents in a family practice residency. Each year an OSCE was administered early in a family practice residency based at the University of North Carolina at Chapel Hill School of Medicine. The OSCE encompassed eight to ten stations, each 15 minutes long: ten minutes for tasks and five minutes for feedback. After the OSCE, focus groups with participating faculty and residents assessed the strengths and weakness of specific stations as well as the overall OSCE structure. Each year the residents' OSCE scores were correlated with their scores on the American Board of Family Physicians In-Training Examination (ITE). Pearson product--moment correlation coefficients were calculated for both the composite ITE scores and the clinical set problem scores. Test reliability was measured by Cronbach's alpha. In the spring of 1995 the faculty completed global evaluations of all the residents in the program at the time, including 16 of the 17 residents who had participated in the OSCEs. The faculty rated the residents in four areas, and a four-point scale was based on the percentile ranges a resident's rating could fall into for each area. Correlation coefficients were calculated for the OSCE scores and the faculty's perceptions of the residents' overall performance. The OSCE scores correlated significantly with the faculty's global assessments of knowledge and clinical skills, but not with in-training examination scores. Concurrent validity and reliability estimates did not support using an OSCE for decisions about the residents' competency. The faculty valued the opportunity to observe basic skills for the whole group of residents; the residents found the experience educational but stressful and valued the immediate performance feedback included at each station. The OSCE is a useful tool for teaching basic clinical skills and for forming initial impressions of interns' clinical styles and abilities, but it should be used with caution for summative assessments.

Background: Medical student learn how to perform eye examinations since beginning in clinical skills laboratory. Objective Structured Clinical Examination (OSCE) was used to assess students’ clinical skills performance every year and to indicate students’ progress. To evaluate medical students’ clinical skills progress, and to evaluate OSCE of eye examination skills. Methods: A cohort study, involving first and third year OSCE results in Clinical Skills Laboratory, Faculty of Medicine Gadjah Mada University from medical students batch 2010 (n:185). The scores of OSCE Year 1 compared to Year 3 using Mann whitney test to explore the progress of students clinical skills performance. To correlate between OSCE scores with clinical cases used in OSCE (such as: visual aquity, visual field, segment anterior and posterior examination) using Pearsoncorrelation to evaluate the assessment process. Result: OSCE results show that medical studens give significant progress on their eye examination performance (median OSCE year 1=66, year 3=80; p=0.00). There is strong correlation between clinical cases during OSCE with students score (p=0.005) that show inequality of difficulty level of cases, particularly for first year OSCE that used more detail check list instrument. Conclusion: Medical students show significant progress on their eye examination skills performance. OSCE process need to be improved, especially on developing clinical cases and check list as a scoring instrument. Keywords: medical students, eye examination, OSCE score

The aim of this study was to evaluate whether the objective structured clinical examination (OSCE) and case presentation (CP) as forms of active assessment were effective measures of overall didactic knowledge and clinical performance in a predoctoral dental curriculum. This evaluation was conducted by statistical analysis of quality points (QP) awarded for didactic and clinical performance, CP grades, and OSCE scores for 185 students at Harvard School of Dental Medicine who graduated during the period 2010-14. As part of the requirements for graduation, each student takes three OSCEs and presents two patient cases. Data for the study were obtained from the Office of the Registrar. The results showed no direct correlation between QP and CP grades and no correlation between CP grades and OSCE scores. However, there was a correlation between OSCE scores and QP. Students with honors-level scores on any of the three OSCEs received significantly more QP than students who did not receive honors. In addition, students with passing scores on OSCEs 2 and 3 received significantly more QP than students with failing or marginal OSCE scores. Innovative formats of active assessment such as OSCEs and CPs can promote a student-centered learning environment. These data indicated that, within this study population, there was a positive association between OSCE scores and clinical and didactic performance, supporting the value of OSCEs as a means of assessment.

Effective communication between doctors and patients is a central requirement for good clinical practice. The doctor's interpersonal skills contribute to patient satisfaction and compliance and can positively influence health outcomes [1]. The Royal College of Anaesthetists seeks to assess candidates' communication skills through simulated patient interviews at the primary FRCA objective structured clinical examination (OSCE). We examined the relationship between the objective OSCE mark and the subjective opinions of the participants in the interview. Candidates at the Coventry Primary FRCA Viva/OSCE course take part in mock OSCE examinations which include communication skills stations in which professional actors play the role of patients. After each such station, both the actor and the candidate were asked to complete a proforma giving their opinions of the candidates' communication skills during the interview. A total of 32 doctor:patient encounters were recorded. These were converted to percentages and compared with each other and with the OSCE mark. Comparisons showed that there was no correlation between the OSCE score and either the patient score or the candidate score suggesting that the OSCE score was a poor predictor of simulated patient satisfaction with the interpersonal and communication skills of the doctor. Mean OSCE marks were 87%, exceeding the mean scores given by simulated patients of 65% and by candidates of 77%. The OSCE scores were compared to the simulated patient scores using the paired t-test. This showed that OSCE marks significantly exceeded the simulated patient score (p < 0.0001). The mean scores given by candidates exceeded those given by actors for all criteria measuring communication skills and satisfaction. These results suggest that the OSCE station fails adequately to assess patient satisfaction with doctors' communication skills. The results also demonstrate that candidates have a falsely high opinion of how effectively they are communicating with patients. In training anaesthetists we must stress the importance of communication and there may be an argument for subjective patient input in the scoring of communication skills stations.