Objective Structured Clinical Assessment Research Articles

Development and validation of a new satisfaction scale for objective structured clinical assessments (S-OSCA): A multicenter cross-sectional study

BackgroundNursing clinical competence assessment has acquired a special relevance at the undergraduate and postgraduate levels in recent years. In this context, the Objective Structured Clinical Assessment (OSCA) has emerged as a valid and feasible method of assessing nursing competence. The Satisfaction with Nursing Skill Examination: Objective Structured Clinical Assessment (SINE-OSCA) scale is a valid and reliable 10-item measure that has been developed to evaluate nursing students' satisfaction with the OSCA in the Australian context. Given the importance that OSCA has gained in Spain, it is necessary to validate this tool to be used in one of the most spoken languages in the world. ObjectivesThe purpose of the study was to carry out a modification of the SINE-OSCA, cross-cultural adaptation and a psychometric analysis of the new S-OSCA with Spanish nursing students. DesignA multicenter study of questionnaire development and validation was carried out in 2023 in four Spanish university nursing centers. The study was carried out in 3 phases: design, pilot implementation, and construct validation. Participants/settingThe total population of students from these centers amounted to 1350 students. The final sample consisted of 364 nursing students, selected by convenience sampling. MethodsThe process of translation and cultural adaptation of SINE-OSCA to the Spanish population was carried out following the guidelines proposed by Beaton et al. Content validation, Internal consistency and temporal reliability were evaluated. ResultsThe S-OSCA presents values in the psychometric indicators (V AIKEN, Bland-Altman diagram, and IVC Lawshe) that exceed the cut-off values established even considering the lower limit of the confidence intervals. This spanish version of the SINE-OSCA has a Cronbach's alpha value that is slightly higher than that reported for that original version (0.928 CI 95 % (0.913–0.94)). Regarding temporal reliability, the S-OSCA scale was completed in 40 nursing students at two times separated by an interval of 15 days. The Intraclass Correlation Coefficient (ICC) obtained was 0.974 CI 95 % (0.952–0.986). ConclusionsThe S-OSCA instrument proves to be robust enough to guarantee the quality of its results up to 15 days post-OSCA.

COMPARISON OF THE EFFECTS OF THE IN SITU SIMULATION AND SIMULATED CLINICAL IMMERSION METHODS ON CANCER NURSING STUDENTS’ EXTRAVASATION MANAGEMENT: RANDOMIZED CONTROLLED TRIAL

In situ simulation is thought to offer a good learning environment due to the fact that it is carried out in real clinical practice. This study was performed to compare the effect of in situ simulation and simulated clinical immersion environment on cancer nursing students extravasation management. The study was carried out with a randomized controlled pretest-posttest comparative design. The participants were randomly assigned to either the simulated clinical immersion group (n=19) or the in situ simulation group (n=24). The study data were collected using Extravasation Management Knowledge Test, Objective Structured Clinical Assessment, Students’ Satisfaction and Self-Confidence Scale and the State Anxiety Inventory. The pre-posttest mean scores for the knowledge level and anxiety mean scores of the participants in the simulated clinical immersion group and the in situ group were not statistically significant (p˃.05). The satisfaction and self-confidence scale mean scores and the skill levels of the students in both settings were similar after the implementation (p˃.05). The students' knowledge, skills, anxiety, self-confidence and satisfaction were similar in both simulation-based learning environments. In situ simulation can be recommended for use in nursing education. Considering cost-effectiveness and resources, both simulation methods can be used.

Midwife Educators’ perceptions of the efficacy of the Objective Structured clinical assessment of life-saving interventions - a qualitative interview study in Bangladesh

IntroductionObjective Structured Clinical Assessment (OSCA) is a way of evaluating students or clinicians in how they are carrying out their duties. The aim of this study was to examine how midwifery educators in Bangladesh perceived using OSCA as an assessment device in midwifery education for student performance in life-saving midwifery interventions. MethodIndividual interviews were conducted with 47 academic midwives and clinical midwives using purposive sampling at 38 education institutions in Bangladesh. Content analysis inspired by Elo and Kyngas was used to analyze the data. ResultsThe ability of students to perform effectively in the OSCA-evaluated simulation of life-saving skills was related to the educators’ understanding of the concept of midwifery. The overarching main category of this study showed that for midwifery educators to be able to effectually teach professional, evidence-based midwifery, they need to be able to synthesize the delivery of practical and theoretical skills with pedagogical skills and knowledge. To implement the OSCA tool more effectively, midwifery educators need to understand the underpinning principles of midwifery values and philosophy including leadership, ownership, responsibility, and personal engagement. ConclusionThere is potential to improve the efficacy of using OSCA to deliver the teaching of life-saving skills. Team sessions with midwives and physicians aiming to practice teamwork and role divisions in life-saving interventions are recommended.

Development of objective structured clinical examination-based assessment methods in drug information services lectures

Background: Research has been carried out on the development of objective structured clinical examination OSCE-based assessment methods in Drug Information Services lectures. Objective: This study aims to prepare the assessment, and teaching-learning methods to improve student competence. Method: This research was conducted in three stages, 1) development of the assessment method; 2) development of the syllabus for Drug Information Services lectures; and 3) trial of the assessment method. For the assessment method and syllabus, validation was carried out by experts based on rubric evaluation. As a benchmark for the acceptance of a limited trial, the value is 56.00. The research subjects were 36 undergraduate students of Sanata Dharma University (SDU). Result: The results of the study show that: 1) the assessment method and syllabus have been declared valid by the experts; 2) a limited trial of the assessment method, resulting in an assessment range between 56.39- 75.69. Conclusion: Based on this research, a valid assessment and teaching-learning method has been produced, and in line with the quality of OSCE-based examination as a national standard, so that they can be used in the next lecture in the 2020/2021 term.

The effect of using standardized patients in the Simulated Hospital Environment on first-year nursing students psychomotor skills learning

BackgroundThe aim of this study was to determine the effect of using standardized patients in the Simulated Hospital Environment on first-year nursing students' psychomotor skills. MethodsThis semi-experimental study was carried out in the one-group pretest-posttest design. Students performed the psychomotor skills in the Non-Simulated Clinical Skill Laboratory and Simulated Hospital Environment. The skill performances of the students were evaluated with the Objective Structured Clinical Assessment form. At the same time, students' views on both environments were received. ResultsStudents' mean Objective Structured Clinical Assessment score in the Simulated Hospital Environment (69.42 ± 10.94) was significantly higher than their mean score in the Non-Simulated Clinical Skill Laboratory (57.07 ± 10.05) (p < 0.05). The differences in scores were particularly significant in the stages of meeting the patient, performing the procedure in accordance with the steps and ending the procedure. The majority of the students stated that they could not do some applications in the Non-Simulated Clinical Skill Laboratory in a similar way to the hospital clinic (85.9%), and they felt like they were in a real hospital clinic in the Simulated Hospital Environment (100%). ConclusionsThe results showed that students developed psychomotor skills more fully in a learning environment designed in a similar way to a real hospital clinic setting.

End-of-placement failure rates among clinical psychology trainees:exceptional training and outstanding trainees or poor gate-keeping?

ABSTRACT Objectives: The objective of this multisite study was to examine failure rates across placements during clinical psychology training. Method: Placement supervisors from nine clinical psychology programmes used the Clinical Psychology Practicum Competencies Rating Scale (CΨPRS) to rate trainees’ competencies at the end of one of four placements (P1, P2, P3, P4). Results: Out of a total of 1449 placements, seventeen (1.17%) placements were failed, with fail-rates reducing as trainees progressed from P1 (2.45%) to P2 (0.77%), P3 (0.33%) and P4 (0.37%). CΨPRS norms used to identify competence as Much Below Standards generated similar fail-rates: 1.04% for all placements; 0.61% for P1; 1.28% for P2; 1.33% for P3; and 1.12% for P4. A larger subgroup (6.97% for all placements; 9.41% for P1; 4.34% for P2; 5.65% for P3 and 7.87% for P4) were also categorised as falling Somewhat Below Standards. The majority (88%) of these “Below Standards” placements received a pass grade from their supervisor. Conclusions: Supervisors tend to pass trainees who fail to meet competence thresholds on the CΨPRS. Placement supervisor assessments are essential but insufficient to ensure trainee competence, and supplemental assessments such as objective structured clinical examinations or competency assessment portfolios are important to complement supervisor assessments. KEY POINTS (1) The study reports end of placement failure rates for clinical psychology trainees across nine training institutions. (2) A number of trainees who received scores below competency thresholds were passed by their placement supervisors. (3) The study raises serious concerns about the sole reliance on placement supervisors’ assessments, and recommends additional independent assessments such as the use of standardised objective structured clinical examinations (OSCE) to ensure safe and competent practice among clinical psychology trainees.

Evaluation of Vaccination Training in Pharmacy Curriculum: Preparing Students for Workforce Needs.

Background: To introduce and evaluate a university vaccination training program, preparing final year Bachelor of Pharmacy (BPharm) and Master of Pharmacy (MPharm) students to administer vaccinations to children and adults in community pharmacy and offsite (mobile and outreach) settings. Methods: Final year BPharm and MPharm students were trained to administer intramuscular vaccinations to adults and children. The education program embedded in pharmacy degree curriculum was congruent with the requirements of the Australian National Immunisation Education Framework. The training used a mix of pedagogies including online learning; interactive lectures; and simulation, which included augmented reality and role play. All pharmacy students completing the program in 2019 were required to carry out pre- and post-knowledge assessments. Student skill of vaccination was assessed using an objective structured clinical assessment rubric. Students were invited to complete pre and post questionnaires on confidence. The post questionnaire incorporated student evaluation of learning experience questions. Results: In both cohorts, student vaccination knowledge increased significantly after the completion of the vaccination training program; pre-intervention and post-intervention mean knowledge score (SD) of BPharm and MPharm were (14.3 ± 2.7 vs. 22.7 ± 3.3; p < 0.001) and (15.7 ± 2.9 vs. 21.4 ± 3.2; p < 0.001) respectively. There was no difference between the BPharm and MPharm in the overall knowledge test scores, (p = 0.81; p = 0.95) pre and post scores respectively. Using the OSCA rubric, all students (n = 52) were identified as competent in the skill of injection and could administer an IM deltoid injection to a child and adult mannequin. Students agreed that the training increased their self-confidence to administer injections to both children and adults. Students found value in the use of mixed reality to enhance student understanding of the anatomy of injection sites. Conclusion: The developed vaccination training program improved both student knowledge and confidence. Pharmacy students who complete such training should be able to administer vaccinations to children and adults, improving workforce capability. Mixed reality in the education of pharmacy students can be used to improve student satisfaction and enhance learning.

Development and psychometric testing of the 10-item satisfaction with Nursing Skill Examination: Objective Structured Clinical Assessment scale

The Objective Structured Clinical Examination (OSCE) or Assessment (OSCA) has traditionally been used in disciplines such as medicine and nursing, to assess students' competence to perform clinical skills safely in a simulated hospital environment. Despite its accepted use, a validated and reliable tool has yet to be developed and tested to assess students' perception of and satisfaction with this mode of assessment. This study developed and tested the psychometric properties of a brief Objective Structured Clinical Examination tool for assessing student perception that could have transferability across health education settings. The study used a cross-sectional survey design. Final year students (n = 727) enrolled in an undergraduate nursing program in Western Sydney completed the 10-item Satisfaction with Nursing Skill Examination: Objective Structured Clinical Assessment (SINE-OSCA) Scale in 2017. Exploratory factor analysis uncovered a one-component structure with component loading that ranged from 0.45 to 0.86. Cronbach's alpha of the SINE-OSCA was 0.91. Socio-demographic group comparisons revealed that respondents who were: i) male (p = 0.003); ii) non-native-born (p < 0.001); iii) non-English-speaking (p < 0.001); and iv) International (p = 0.001), reported higher satisfaction with clinical assessments, as measured by the SINE-OSCA scale. The SINE-OSCA scale demonstrates validity and reliability in identifying students who may have difficulty with this mode of clinical skill assessment.

Factors contributing to undergraduate nursing students' satisfaction with a video assessment of clinical skills

BackgroundClinical skill assessment via Objective Structured Clinical Assessment (OSCA) has many challenges for undergraduate nursing students. These include high levels of anxiety that can compromise performance during the assessment, inconsistency with assessor reliability and is inconsistent with clinical skills performance in the real world. The implementation of a Video Assessment of Clinical Skills (VACS) that integrates formative feedback may be a way to address the challenges posed by OSCA assessment. ObjectivesThe aim of this study was to examine the acceptability, utility, and nursing student satisfaction with a formative feedback strategy – the Video Assessment of a Clinical Skill (VACS). DesignA cross sectional survey. SettingsUndergraduate Bachelor of Nursing degree students from a large Australian University. ParticipantsThird year undergraduate nursing students (final year) enrolled in a Bachelor of Nursing Program. MethodsParticipants were recruited via purposive sampling. A pre-survey (prior to VACs assessment) and post-survey (after VACS assessment) were completed. This paper reports on the open-ended responses in the post-survey that explored students' insights and perceptions into formative feedback and its impact on their learning for the VACS assessment. ResultsA total of 731 open-ended responses were analysed with findings being organised into 3 major themes; (i) Flexibility and reflexivity, (ii) Editing and repeated attempts, and (iii) Working together. ConclusionsVideo Assessment of a Clinical Skill has demonstrated good utility, acceptability, and satisfaction among undergraduate nursing students.

Evaluating the implementation of English communication therapy (ECT): An objective structured clinical assessment (OSCA) approach

This study is aimed at evaluating the implementation of the English Communication Therapy (ECT) instructional program oriented to the Objective Structured Clinical Assessment (OSCA) for the students of the D-3 Nursing Study Program at STIKES Al-Irsyad Al-Islamiyyah, Cilacap. Three research problems are put forward concerning (1) the fit of the program in terms of the variables context, input, process, and product; (2) obstacles faced by students; and (3) attainment of the ECT learning objectives. The study is evaluation research using the qualitative and quantitative CIPP model. The research subjects are 168 students of the Nursing Study Program of STIKES Al-Irsyad Al-Islamiyyah. Data were collected using interviews, questionnaires, and documentation. The findings show that (1) evaluation of the four variables of context, input, process, and product shows that ECT can be stated as in the good or fit category; (2) main obstacles faced by students are lack of vocabulary and low level of self-confidence; and (3) students are able to achieve the passing grade of 74 at each station on the competency of English speaking about nursing matters.

Examining the effect of simulation based learning on self-efficacy and performance of first-year nursing students.

Self-efficacy is an individual's belief in oneself and in one's own judgment and ability to be successful in a task. Strong self-efficacy allows for the achievement of success, well-being, personal development and the diversification of skills. An individual's perception of his/her self-efficacy plays a key role in adopting behaviors and initiating and sustaining change. The first aim of this study is to evaluate the effects of simulation-based learning on the self-efficacy and performance of first-year nursing students. The second aim is to examine students' pre- and post-scenario proficiency in their self-assessment of competence in regard to scenario objectives to compare the students' self-assessment and their instructor's assessment in terms of performance evaluations. This semi-experimental study was conducted with sixty-five first-year nursing students. A standardized patient method was used as a simulation technique. Students' self-efficacy and their perceptions of their own competence in completing skills were measured using the General Self-Efficacy Scale and the Proficiency Assessment Form, respectively. Students' actual performance was evaluated via an Objective Structured Clinical Assessment and performance evaluation checklists. The mean self-efficacy score of students was 52.68 (±10.19) before the scenario and 49.59 (±12.90) post-scenario (p = .001). With regard to their scenario objectives, a decrease was observed in students' proficiency in their post-scenario self-assessment of competence. This decrease was only statistically significant in the "Establishing a safe patient unit" objective (t = 2.27; p = .03). We observed that self-efficacy scores declined in the post-simulation scenario. Using standardized patients in simulation training allows novice nursing students to meet a real patient and to recognize their own true self-efficacy.

Designing a blended training program and its effects on clinical practice and clinical reasoning in midwifery students.

INTRODUCTION:Proper empowerment of medical students in encounter with the complexities of the clinical environment is one of the requirements for health services. In this regard, the development of problem-solving skills, critical thinking is essential for medical education. The purpose of this study was to investigate the effect of designing a blended training program on the practice and clinical reasoning of midwifery students.MATERIALS AND METHODS:In a quasi-experimental study with control group, 41 undergraduate midwifery students were randomly assigned to two intervention and control groups by four blocking. In the first group, the 15-h workshop was held in the traditional teaching method and in the second group in the blended method, about three emergency aspects of midwifery. Data were collected and analyzed through objective structured clinical examination and clinical assessment evaluation before and after the educational intervention.RESULTS:The mean and standard deviation of the age of participants were 23.54 ± 2.19 years. The mean score of clinical practice before the beginning of the study in the integrated training group was 16.68 ± 5.49 and at the end of the study was increased to 35.75 ± 4.54, which was significant between the two groups (P = 0.035). The clinical reasoning score was changed at the beginning of the study in the blended training group of 6.77 ± 3.57 and at the end of the intervention was 11.58 ± 2.83. There was no significant difference between the two groups (P = 0.81).CONCLUSION:The results showed that the use of blended methods in comparison with conventional education can improve clinical practice, but the improvement of students’ ability in clinical reasoning requires more effort.

Facilitating peer based learning through summative assessment - An adaptation of the Objective Structured Clinical Assessment tool for the blended learning environment.

Adapting a course from face to face to blended delivery necessitates that assessments are modified accordingly. In Australia the Objective Structured Clinical Assessment tool, as a derivative from the Objective Structured Clinical Examination, has been used in the face-to-face delivery mode as a formative or summative assessment tool in medicine and nursing since 1990. The Objective Structured Clinical Assessment has been used at Charles Darwin University to assess nursing students' simulated clinical skills prior to the commencement of their clinical placements since 2008. Although the majority of the course is delivered online, students attend a one-week intensive clinical simulation block yearly, prior to attending clinical placements. Initially, the Objective Structured Clinical Assessment was introduced as a lecturer assessed summative assessment, over time it was adapted to better suit the blended learning environment. The modification of the tool from an academic to peer assessed assessment tool, was based on the empirical literature, student feedback and a cross-sectional, qualitative study exploring academics' perceptions of the Objective Structured Clinical Assessment (Bouchoucha etal., 2013a, b). This paper presents an overview of the process leading to the successful adaptation of the Objective Structured Clinical Assessment to suit the requirements of a preregistration nursing course delivered through blended learning. This is significant as many universities are moving their curriculum to fully online or blended delivery, yet little attention has been paid to adapting the assessment of simulated clinical skills. The aim is to identify the benefits and drawbacks of using the peer assessed Objective Structured Clinical Assessment and share recommendations for successful implementation.

Intimate Partner Violence Screening and Counseling: An Introductory Session for Health Care Professionals.

Intimate partner violence is a serious public health concern in the United States. Despite recommendations that physicians should routinely screen their patients, research has shown that lack of specific training has resulted in many health care professionals feeling unable to adequately perform this difficult but vital task. Though many educational resources exist to teach intimate partner violence screening, they often lack specific guidance on how to navigate this difficult conversation. In addition, they often lack formal teaching on how to counsel and refer patients who are victims of intimate partner violence. This unique module, intended for a small-group setting of four to eight students, contains an intimate partner violence checklist with sample language that covers both screening and counseling using a motivational interviewing framework. Additional materials include a checklist companion for tips on how to navigate the conversation, two cases for role-play, a facilitator guide, and an objective structured clinical encounter case and assessment rubric. This module was given to 260 second-year medical students at the Warren Alpert Medical School between 2015 and 2017 and was rated highly by almost 90% of students. After completing this module, learners will be able to appropriately screen for intimate partner violence as well as counsel and refer patients who have screened positive. By implementing this module, educators can increase the number of health care professionals able to broach this difficult conversation with patients who may be in need of help and may otherwise go unaided.

The assessment of medical competencies

ObjectiveTo describe the most widely used tools in the assessment of medical competencies, analyze their prevalence of use, their advantages and disadvantages and propose an appropriate model for our context. MethodsWe conducted a narrative review of articles from MEDLINE, following the PRISM protocol, and analyzed a total of 62 articles. ResultsThe assessment of competencies is heterogeneous, especially in the educational and professional settings. The specific and technical competencies acquired during university education are mainly assessed using the objective structured clinical assessment. In the professional setting, core competencies are assessed using the 360° technique. ConclusionsWe need a rigorous empiric comparison of the efficiency of the tools according to the type of competency. We propose a competency management model for the “undergraduate/graduate/active professional” continuum, whose goal is to improve training and professional practice and thereby increase the quality of patient care.

La evaluación de competencias en Medicina

ObjectiveTo describe the most widely used tools in the assessment of medical competencies, analyse their prevalence of use, their advantages and disadvantages and propose an appropriate model for our context. MethodsWe conducted a narrative review of articles from MEDLINE, following the PRISM protocol, and analysed a total of 62 articles. ResultsThe assessment of competencies is heterogeneous, especially in the educational and professional settings. The specific and technical competencies acquired during university education are mainly assessed using the objective structured clinical assessment. In the professional setting, core competencies are assessed using the 360° technique. ConclusionsWe need a rigorous empiric comparison of the efficiency of the tools according to the type of competency. We propose a competency management model for the «undergraduate/graduate/active professional» continuum, whose goal is to improve training and professional practice and thereby increase the quality of patient care.

Teaching the basics of echocardiography in the undergraduate: Students as mentors

ObjectiveTo analyze the ability of medical students to incorporate the practical teaching of basic echocardiography planes using a peer mentoring design. MethodologyThirty-six medical students previously trained in obtaining echocardiography planes (mentors) taught the other 5th-year students (n=126). The teaching methodology included three stages: theory (online course), basic training (three 15h sessions of practical experience in ultrasound and at least 20 echocardiographic studies per mentor) and objective structured clinical assessment (OSCA), which scored the appropriateness of the basic ultrasound planes and the correct identification of 16 cardiac structures. ResultsThe students’ weighted mean score in the OSCA was 8.66±1.98 points (out of 10). Only 10 students (8.4%) scored less than 5, and 15 (12.6%) scored less than 7. Fifty students (42%) scored 10 points. The most easily identified structure was the left ventricle in the short-axis parasternal plane, with 89.9% of correct answers. The most poorly identified structure was the mitral valve in the subxiphoid plane, with 69.7% of correct answers. ConclusionsPeer mentoring-based teaching achieves an appropriate level of training in obtaining basic echocardiography planes. The training period is relatively short. The peer mentoring system can facilitate the implementation of teaching on basic aspects of ultrasound to a large number of undergraduate students.

Enseñanza de las bases de la ecocardioscopia en el pregrado: los estudiantes como mentores

ObjectiveTo analyse the ability of medical students to incorporate the practical teaching of basic echocardiography planes using a peer mentoring design. MethodologyThirty-six medical students previously trained in obtaining echocardiography planes (mentors) taught the other 5th-year students (n=126). The teaching methodology included three stages: theory (online course), basic training (three 15h sessions of practical experience in ultrasound and at least 20 echocardiographic studies per mentor) and objective structured clinical assessment (OSCA), which scored the appropriateness of the basic ultrasound planes and the correct identification of 16 cardiac structures. ResultsThe students’ weighted mean score in the OSCA was 8.66±1.98 points (out of 10). Only 10 students (8.4%) scored less than 5, and 15 (12.6%) scored less than 7. Fifty students (42%) scored 10 points. The most easily identified structure was the left ventricle in the short-axis parasternal plane, with 89.9% of correct answers. The most poorly identified structure was the mitral valve in the subxiphoid plane, with 69.7% of correct answers. ConclusionsPeer mentoring-based teaching achieves an appropriate level of training in obtaining basic echocardiography planes. The training period is relatively short. The peer mentoring system can facilitate the implementation of teaching on basic aspects of ultrasound to a large number of undergraduate students.

Assessment of the Intrinsic CanMEDS Roles in Diagnostic Radiology Residents using an Objective Structured Clinical Assessment (OSCE)

This article was migrated. The article was not marked as recommended. Purpose: Non-Medical Expert or Intrinsic CanMEDS Roles, as outlined by the Royal College of Physicians and Surgeons of Canada, can be difficult to evaluate in an objective and specialty-specific manner. This study investigates an Objective Structured Clinical Examination (OSCE) evaluation tool to assess these competencies for Diagnostic Radiology Residents. Methods: A five station CanMEDS OSCE was developed for postgraduate year 3 and 4 Residents to evaluate the Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional Roles. Performance was assessed by postgraduate year 5 Residents using standardized scoring rubrics. CanMEDS OSCE scores were correlated with American College of Radiology (ACR) scores and Medical Expert OSCEs. Results: Seventy Residents in three separate cohorts (n=21, 26, 23) participated in the CanMEDS OSCE. Mean station scores were consistent across cohorts. In general, one-way ANOVAs showed no effect of postgraduate year on station scores. There were no significant correlations between CanMEDS OSCE scores and ACR exam scores or CanMEDS OSCE scores and Medical Expert OSCE scores, demonstrating divergent construct validity. In turn, this indicates construct validity for the CanMEDS OSCE by demonstrating that unique competencies are being measured. In contrast, there was a correlation between ACR exam scores and Medical Expert OSCE scores, confirming that these both assess the same construct. Conclusions: An OSCE can be a useful assessment tool to assess Intrinsic CanMEDS Roles in a specialty-specific manner. Correlational analyses indicate that unique competencies are being evaluated that are not captured by other traditional assessment means.

Validity and Feasibility Evidence of Objective Structured Clinical Examination to Assess Competencies of Pediatric Critical Care Trainees.

The purpose of this study was to provide validity and feasibility evidence for the use of an objective structured clinical examination in the assessment of pediatric critical care medicine trainees. This was a validation study. Validity evidence was based on Messick's framework. A tertiary, university-affiliated academic center. Seventeen pediatric critical care medicine fellows were recruited in 2012 and 2013 academic year. None. All subjects completed an objective structured clinical examination assessment. Seventeen trainees were assessed. Simulation scenarios were developed for content validity by pediatric critical care medicine and education experts using CanMEDS competencies. Scenarios were piloted before the study. Each scenario was evaluated by two interprofessional raters. Inter-rater agreement, measured using intraclass correlations, was 0.91 (SE = 0.09) across stations. Generalizability theory was used to evaluate internal structure and reliability. Reliability was moderate (G-coefficient = 0.67, Φ-coefficient = 0.52). The greatest source of variability was from participant by station variance (40.6%). Pearson correlation coefficients were used to evaluate the relationship of objective structured clinical examination with each traditional assessment instruments: multisource feedback, in-training evaluation report, short-answer questions, and Multidisciplinary Critical Care Knowledge Assessment Program. Performance on the objective structured clinical examination correlated with performance on the Multidisciplinary Critical Care Knowledge Assessment Program (r = 0.52; p = 0.032) and multisource feedback (r = 0.59; p = 0.017), but not with the overall performance on the in-training evaluation report (r = 0.37; p = 0.143) or short-answer questions (r = 0.08; p = 0.767). Consequences were not assessed. Validity and feasibility evidence in this study indicate that the use of the objective structured clinical examination scores can be a valid way to assess CanMEDS competencies required for independent practice in pediatric critical care medicine.