Final Objective Structured Clinical Examination Research Articles

An evaluation of the inter-rater reliability in a clinical skills objective structured clinical examination

Background. An objective structured clinical examination (OSCE) is a performance-based examination used to assess health sciences students and is awell-recognised tool to assess clinical skills with or without using real patients.Objectives. To determine the inter-rater reliability of experienced and novice assessors from different clinical backgrounds on the final mark allocationsduring assessment of third-year medical students’ final OSCE at the University of the Free State.Methods. This cross-sectional analytical study included 24 assessors and 145 students. After training and written instructions, two assessors per station(urology history taking, respiratory examination and gynaecology skills assessment) each independently assessed the same student for the same skill bycompleting their individual checklists. At each station, assessors could also give a global rating mark (from 1 to 5) as an overall impression.Results. The urology history-taking station had the lowest mean score (53.4%) and the gynaecology skills station the highest (71.1%). Seven (58.3%) ofthe 12 assessor pairs differed by >5% regarding the final mark, with differences ranging from 5.2% to 12.2%. For two pairs the entire confidence interval(CI) was within the 5% range, whereas for five pairs the entire CI was outside the 5% range. Only one pair achieved substantial agreement (weightedkappa statistic 0.74 ‒ urology history taking). There was no consistency within or across stations regarding whether the experienced or novice assessorgave higher marks. For the respiratory examination and gynaecology skills stations, all pairs differed for the majority of students regarding the globalrating mark. Weighted kappa statistics indicated that no pair achieved substantial agreement regarding this mark.Conclusion. Despite previous experience, written instructions and training in the use of the checklists, differences between assessors were found inmost cases.

Objective structured clinical examinations (OSCEs) for students’ training and assessment in the French respiratory medicine departments in 2021: An overview

BackgroundMedical professional performances can be assessed by objective structured clinical examinations (OSCEs) where medical trainees go through a series of simulated clinical situations. OSCEs are now the gold standard for the assessment of medical students’ competence during their training. In France, the first national OSCEs will take place in May 2024 and respiratory teachers will be involved in this reform and will use OSCEs for students’ training and assessment in respiratory medicine. Students training regarding this final OSCE may vary across medical faculties and may impact students' results. Therefore, we aimed to provide a national overview of OSCE's training performed by respiratory teachers and their interest in developing a common French databank of OSCEs. MethodsWe conducted a national anonymous online survey among the members of the French college of respiratory teachers (CEP), from 2021 February the 15th to 2021 June the 15th. The survey consisted of 32 questions. ResultsAmong 118 French pulmonologists teachers, 52 (45%) responded to the survey. We received a response from at least one of each of the French Medical Universities. Twenty-two (42%) had received specific training on how to conduct an OSCE. Twenty-eight (54%) of respondents used OSCEs for training purposes and 24 (46%) for assessment purposes, for less than 1 year in more than half of the participants. The average satisfaction scores out of 10 about OSCEs was 7.3 ± 1.7 for training and 7.4 ± 1.5 for students’ assessment. Respondents were willing (8.9 ± 1.8 out of ten) to develop a common databank to share OSCEs subjects in respiratory medicine in France. ConclusionsThis survey confirms heterogeneity in the training and the use of OSCEs among French respiratory teachers. However, a common national databank could be a useful tool to reduce these disparities.

Influence of Clinical Expertise Between Clinician-Instructors Versus Student-Instructors on the Effectiveness of Ultrasound Courses

To investigate how the extent of an ultrasound instructor's clinical expertise influences the level of hands-on ultrasound competency achieved by clinicians after three-day ultrasound courses in abdominal and emergency ultrasound. The second goal was to determine how physicians in residency rate the sonographic and didactic skills of student instructors compared to medical staff instructors. A total of N = 100 residents participating in a 3-day ultrasound workshop were randomly trained either by 15 student instructors or by clinical staff physicians, both with > 2years of teaching experience. Both instructor groups had previously completed 120 hours of didactic instructor training. Ultrasound competencies were assessed by a standardized objective structured clinical examination (OSCE) of healthy individuals. The sonographic and didactic skills of both instructor groups were rated by questionnaires with six-step Likert items (1 = excellent, 6 = poor). The cohort, trained by student instructors, achieved the same scoring level as the second cohort, trained by physician instructors (mean score 76.4, versus 73.7 of max.100points, p = 0.28) in the final OSCE. The sonographic qualification was rated for student educators as 1.44 (mean) [1.1-1.62], versus residents 1.46 [1.26-1.61], versus attendings 1.29 [1.05-1.61]. Didactic skill levels were rated without significant differences with means of 1.53, 1.57 and 1.53, respectively. The didactic competence of student instructors does not differ statistically from the competency levels of experienced physicians. In terms of the acquired ultrasound skills of trainees, our data indicate that student instructors can be as efficient as staff instructors. Therefore, student instructors can be employed as resource-saving ultrasound educators without decreasing the skills level achieved by course participants, provided that they previously underwent a comprehensive didactic and sonographic training program.

Peer-assisted Learning: Clinical Skills Training for Pharmacy Students

Objective. To assess the impact of peer-teaching on student scores and confidence when preparing for a final objective structured clinical examination (OSCE) within a Doctor of Pharmacy program. Methods. First-year pharmacy students (n=45) attended a peer-led training session led by upperclassmen (n=17) on a variety of clinical skills to be assessed on a final course OSCE. Their scores were collected and compared to students who did not attend the training. Confidence scores were also evaluated using voluntary pre- and post-surveys. Results. An overall 3% increase in scores was recorded from the objective skills examination. Student confidence scores also increased for each of the skills evaluated with an overall improvement of 1.1 on a 5-point Likert scale. Conclusion. Peer-assisted learning was effective in increasing student performance and confidence in the OSCE. Based on the positive results, the peer-led training event will be improved upon and used again in the future.

Validity of In-Training Evaluation Report (ITER) in Saudi Pediatric Residency Training. Single center experience

This article was migrated. The article was not marked as recommended. Objective: The study aimed to examine validity evidence of in-training evaluation report (ITER) at King Abdulaziz University Hospital (KAUH)'s pediatric residency program. Predictive validity of ITERs, to assess learner acquisitions of required competencies were studied. We used ITER scores to identify trainees with unprofessional conduct. Methods: One-year data for 35 residents in pediatric residency training program at KAUH were reviewed. Data were extracted from 360 ITERs completed by faculty members and specialists. ITERs' reliability was evaluated using Cronbach's alpha. ITERs scores were correlated with final Objective Structured Clinical Examinations (OSCE) and written exams scores. Professionalism composite scores from ITERs were compared with scores of trainees reported for unprofessional attitude. Results: Specialists had high variability in scoring competencies with average standard deviations (SD) of 0.85 compared with 0.15 by faculty. Reliability of ITER scores using Cronbach's alpha was 0.895. Good correlation was observed between competencies ranging between 0.86 to 0.29. There was fair correlation between ITER composite scores and those of final OSCE and written exams. There was no correlation between professionalism scores and residents reported for unprofessional conduct. Conclusion ITER scores had good validity and reliability despite variability observed between faculty and specialist when scoring competencies. ITERs failed to detect trainees with unprofessional conduct.

Introduction of Objective Structured Clinical Examination, an assessment method for undergraduate students in a dental school

Objective: Undergraduate education in a dentistry program aims to produce competent practitioners equipped with appropriate knowledge, behavior, attitudes and skills necessary to blend into independent dental practice. Assessment is crucial in the learning process of the students, and assessment drives learning is universally accepted. A number of traditional assessment methods are in common use; lately, Objective Structured Clinical Examination (OSCE) has been suggested to assess the clinical competencies and skills in health sciences. Argyris recommended developing an implementation strategy of advocacy and inquiry to encourage the acceptance of personal responsibility to overcome resistance (to changes being introduced into the existing assessment system). The purpose of this study was to investigate the effectiveness of implementation strategy by measuring the attitudes of both staff and students towards OSCE in a dental school. Methods: Following sequential steps of providing information, advocacy, participation to minimize resistance towards introduction of OSCE in the course of oral medicine, fourteen staff members and thirty six students participated in pilot/mini OSCE which included one rest and five test stations. Later, a more definitive final OSCE was designed with 10 test plus one rest station. After completion of both pilot/mini and final OSCE, a questionnaire adopted from a previously conducted study was answered by the staff and students. Results: The staff scores were significantly higher following the final OSCE for all the questions except “increase use”. In the student group, “increase use” and “difficulty/ease” improved from pilot to final OSCE, but the difference was statistically insignificant. Comparison between the groups revealed staff were significantly more enthusiastic, believed the use of OSCE to be increased and felt the test was relevant. Conclusion: Involvement, participation and joint control of the staff in designing and developing the OSCE stations and its implementation led to a successful implementation of the new assessment method.

Can near-peer medical students effectively teach a new curriculum in physical examination?

BackgroundStudents in German medical schools frequently complain that the subject ‘clinical examination’ is not taught in a satisfying manner due to time constraints and lack of personnel resources. While the effectiveness and efficiency of practice-oriented teaching in small groups using near-peer teaching has been shown, it is rarely used in German medical schools. We investigated whether adding a new near-peer teaching course developed with student input plus patient examination under supervision in small groups improves basic clinical examination skills in third year medical students compared to a traditional clinical examination course alone.MethodsThird year medical students registered for the mandatory curricular clinical examination course at the medical faculty of the Technische Universität München were invited to participate in a randomised trial with blinded outcome assessment. Students were randomised to the control group participating in the established curricular physical examination course or to the intervention group, which received additional near-peer teaching for the same content. The learning success was verified by a voluntary objective structured clinical examination (OSCE).ResultsA total of 84 students were randomised and 53 (63%) participated in the final OSCE. Students in the control group scored a median of 57% (25th percentile 47%, 75th percentile 61%) of the maximum possible total points of the OSCE compared to 77% (73%, 80%; p < 0.001) for students in the intervention group. Only two students in the intervention group received a lower score than the best student in the control group.ConclusionAdding a near-peer teaching course to the routine course significantly improved the clinical examination skills of medical students in an efficient manner in the context of a resource-constrained setting.

Can training in musculoskeletal examination skills be effectively delivered by undergraduate students as part of the standard curriculum?

There is a need to improve competence of musculoskeletal system (MSS) examination in medical students and junior doctors. Peer-assisted learning (PAL) is a technique whereby students learn from and with each other. This study aimed to determine whether PAL can be integrated into standard undergraduate medical curricula to improve MSS examination using the gait, arms, legs, spine (GALS) screening tool. Fifty final-year students (trainers) were trained using GALS for MSS examination while attending a standard clinical medical attachment at Glasgow Royal Infirmary. These students delivered GALS training to a further 159 students (trainees). Pre/post-confidence questionnaire (100-mm visual analogue scale) and written feedback were obtained. Final Objective Structured Clinical Examination (OSCE) scores from an MSS station were compared with a control group of 229 students randomized to other hospitals for the standard MSS training. Analysis of completed trainer questionnaires (30/50) showed increased confidence in all parts of GALS after training [<47 (19) cf. >88 (12); P < 0.005]. Similarly, confidence in trainees (136/159) who answered the questionnaire increased [<43 (19) cf. >85 (15); P < 0.005]. Written comments highlighted that students would recommend PAL. OSCE results showed 84% (192/229) of students in the control group passed the MSS station, with 87% (139/159) of trainees (P = 0.3) and 100% (50/50) of trainers (P < 0.01). MSS examination skills are improved by integrating PAL into the undergraduate medical curriculum, with student confidence being increased, and higher OSCE scores.

Development of a tool to support holistic generic assessment of clinical procedure skills

The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic. This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills. We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff. Results In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was > 70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station. This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).

An implementation strategy for introducing an OSCE into a dental school

The Objective Structured Clinical Examination (OSCE) uses a series of test-stations to test clinical competencies. The introduction of an OSCE in a dental school is always a new experience for both staff and students and may result in a change in assessment methods. As resistance could develop when changes are introduced into an organisation, the use of a strategy for the implementation of such change will help to diminish opposition and may therefore result in the co-operation of staff and their departments. The objective of this study was to investigate the effectiveness of an implementation strategy by measuring attitudes of both staff and students towards the OSCE as a new form of clinical assessment in a dental school (ACTA). 'Stepwise' behaviour change (with information, participation and commitment as tools) was used as a strategy to minimise protective behaviour to the introduction of an OSCE. After lectures on assessment, 59 staff members participated in a mini-OSCE with eight test-stations, playing both the role of a student and observer. A questionnaire, designed to test attitudes and commitment towards the new OSCE was completed after the examination. Six months later, 22 staff of all departments had developed and run a pilot OSCE for 44 students. A similar questionnaire was answered by staff and students. A year later, another OSCE for all 103 third year students was designed, organised and evaluated with full co-operation of the clinical teaching staff. Staff total attitude grew positively (P = 0.001). Student's total attitude was lower than staff (P < 0.001) The results of the survey after the mini-staff-OSCE and pilot and final OSCE were favourable in terms of the acceptance of use of an OSCE for the assessment of clinical competences. The implementation strategy appears to have been successful. The objective of gaining the co-operation of staff and departments and avoiding resistance to change was achieved.