Can cultural competency self-assessment predict OSCE performance?

Abstract

Context and setting Although cultural competency training is required by various accreditation bodies in medical education, there is no agreement upon how best to evaluate such training programmes. According to systematic reviews, most studies evaluating cultural competency training of health professionals utilise self-assessments. Few studies have employed objective structured clinical examinations (OSCEs). No study has reported the relationship between self-assessments and OSCEs in evaluating cultural competency training. Why the idea was necessary The OSCE is generally regarded as a better method of measuring competency in clinical skills than self-assessment. However, conducting an OSCE is so much more demanding than administrating a self-efficacy survey that the latter is often chosen as the means to measure cultural competency skills. This study addresses the unanswered question of whether self-assessment can substitute for the OSCE in measuring cross-cultural communication skills. What was done We recruited 57 Year 5 students at our medical school to participate in the study between January 2006 and June 2006. Students filled out a survey containing the Inventory for Assessing the Process of Cultural Competence among Healthcare Professionals-Revised (IAPCC-R), which has 5 subscales, the California Brief Multicultural Competence Scale (CBMCS), which has 4 subscales, and a survey designed to measure US residents’ preparedness to deliver cross-cultural care (CCC), which has 2 subscales. The Cronbach’s alpha coefficients for the 3 scales were 0.71, 0.84 and 0.92, respectively. All students were then evaluated with an objective structured clinical examination (OSCE) which tested their ability to explore sociocultural factors influencing a standardised patient adherence to chronic disease treatment. Multiple regressions were conducted to predict the patient perspectives (PP) and social factors (SF) subscales of the OSCE. All 11 subscales of the IAPCC-R, CBMCS and CCC were selected with the stepwise method. We analysed the data using SAS Version 9.1. Evaluation of results and impact In the PP regression model, PP was predicted by the ‘skill’ subscale of the IAPCC-R and the ‘awareness’ subscale of the CBMCS. Overall model test was significant (F[2,56] = 4.59, P = 0.0145, R2 = 0.15) and parameter estimates were significant at a level of 0.05. The estimated model was PP = 1.32 – 0.18 × skill + 0.19 × awareness. In the SF regression model, SF was predicted by the ‘encounter’ subscale of the IAPCC-R and the ‘awareness’ subscale of the CBMCS. Overall model test was significant (F[2,56] = 4.45, P = 0.0163, R2 = 0.14), but only the parameter estimate of awareness was significant. The estimated model was SF = 0.43 – 0.25 × encounter + 0.27 × awareness. In addition, all of the assumptions including equal error variance, normality of error terms, influence cases and multi-co-linearity were checked. In summary, although a few of the self-assessment subscales predicted OSCE performance, the effects were not strong. Our results suggest that medical students may not be able to accurately assess their own cross-cultural communication skills, and that the current self-efficacy scales cannot substitute for the OSCE to measure cultural competency. Future studies on cultural competency training evaluation should look at the development of better survey instruments which can be triangulated with an OSCE.