Medical Council Of Canada Qualifying Examination Research Articles

Conducting an objective structured clinical examination under COVID-restricted conditions

BackgroundThe administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. The purpose of this study was to gather evidence to support the reliability and validity of the modified NAC Examination.MethodsThe modified NAC Examination was delivered to 2,433 candidates in 2020 and 2021. Cronbach’s alpha, decision consistency, and accuracy values were calculated. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs. female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs. non-CSA), postgraduate training (PGT) (no PGT vs. PGT), and language of examination (English vs. French). Criterion relationships were summarized using correlations within and between the NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores.ResultsReliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were statistically significant. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40).ConclusionsThe NAC Examination yields reliable total scores and pass/fail decisions. Expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that required modifications due to COVID.

Medical Council of Canada abandons controversial exam.

The Medical Council of Canada (MCC) has scrapped a controversial medical licensing exam after repeated cancellations and technical difficulties during the pandemic. In a June 10 statement, the organization confirmed it is “stepping away” from the MCC Qualifying Examination (MCCQE) Part II,

Does Pass/Fail on Medical Licensing Exams Predict Future Physician Performance in Practice? A Longitudinal Cohort Study of Alberta Physicians

ABSTRACT The purpose of this longitudinal study was to gather extrapolation evidence of validity by assessing whether performance on a national medical licensing exam, in addition to practice and socio-demographic variables, is predictive of future physician performance in practice. The study focused on a cohort of 3,404 physicians who were registered with the College of Physicians and Surgeons of Alberta (CPSA) and who completed the Medical Council of Canada Qualifying Examination (MCCQE) Parts I and II between 1992–2017. Separate multivariate quasi-Poisson regression models were run to assess the degree of relationship between first-time pass/fail status on the MCCQE I and II, and several CPSA socio-demographic variables and several CPSA socio-demographic variables, in addition to complaints/physician and various prescribing flags. Candidates who failed the MCCQE I on their first attempt had 27% more complaints lodged against them, compared to those who passed. Physicians who failed the MCCQE II on their first attempt prescribed 2+ benzodiazepines and 2+ opioids to 30% more patients than those who passed. Conclusions: Performance on the MCCQE Part I and II is an important predictor of physician performance. Combined with other critical variables, these measures provide important evidence to aid in risk modeling efforts and to guide educational interventions for physicians at an early stage of their careers.

Does Emotional Intelligence at medical school admission predict future licensing examination performance?

BackgroundMedical school admissions committees are seeking alternatives to traditional academic measures when selecting students; one potential measure being emotional intelligence (EI). If EI is to be used as an admissions criterion, it should predict future performance. The purpose of this study is to determine if EI scores at admissions predicts performance on a medical licensure examinationMethodsAll medical school applicants to the University of Ottawa in 2006 and 2007 were invited to complete the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT v2.0) after their interview. Students were tracked through medical school into licensure and EI scores were correlated to their scores on the Medical Council of Canada Qualifying Examination (MCCQE) attempted between 2010 and 2014.ResultsThe correlation between the MSCEIT and the MCCQE Part I was r (200) = .01 p =. 90 The covariates of age and gender accounted for a significant amount of variance in MCCQE Part I scores (R2 = .10, p<.001, n=202) but the addition of the MSCEIT scores was not statistically significant (R2 change = .002, p=.56). The correlation between the MSCEIT and the MCCQE Part II was r(197) = .06, p = .41. The covariates of age and gender accounted for some variance in MCCQE Part II scores (R2 = .05, p = .007, n=199) but the addition of the MSCEIT did not (R2 change = .002 p =.55).ConclusionThe low correlations between EI and licensure scores replicates other studies that have found weak correlations between EI scores and tests administered at admissions and during medical school. These results suggest caution if one were to use EI as part of their admissions process.

Does the Medical College Admission Test (MCAT) predict licensing examination performance in the Canadian context?

Research on the predictive validity of the Medical College Admissions Test (MCAT) on licensing examination performance is varied in its conclusions, with only a few studies examining this relationship in a Canadian context. We assessed the predictive validity of the MCAT on successful performance on the Medical Council of Canada Qualifying Examination (MCCQE) Part 1 by students attending the Cumming School of Medicine. Prospective data were collected on MCAT score and sub-section scores, MCCQE decision, multiple mini interview (MMI) performance, gender, and age. The cohort was divided into a derivation cohort (2013 and 2014) and validation cohort (2015 and 2016). Students were dichotomized into pass or fail on MCCQE. Multiple logistic regression in which our dependent variable was MCCQE Part I examination success at the first attempt was used, and potential explanatory variables were age, gender, MCAT total score, and sub-scores for the biological sciences (MCAT-BS), physical sciences, and verbal reasoning, GPA, and MMI ratings. For the derivation cohort MCAT-BS was associated with success on the MCCQE Part I. The odds ratio for this association of 1.37 (95% confidence interval [1.01, 1.85], p = 0.04). When we applied the MCAT-BS to our validation cohort the odds ratio of MCCQE Part I examination success was 1.42 [1.10, 1.83], p = 0.007) and the area under the ROC curve was 0.66 [0.54, 0.79]). The MCAT-BS predicted successful performance on the MCCQE Part 1 Examination in the Canadian setting.

CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores.

Typically, only a minority of applicants to health professional training are invited to interview. However, pre-interview measures of cognitive skills predict for national licensure scores (Gauer et al. in Med Educ Online 21 2016) and subsequently licensure scores predict for performance in practice (Tamblyn et al. in JAMA 288(23): 3019-3026, 2002; Tamblyn et al. in JAMA 298(9):993-1001, 2007). Assessment of personal and professional characteristics, with the same psychometric rigour of measures of cognitive abilities, are needed upstream in the selection to health profession training programs. To fill that need, Computer-based Assessment for Sampling Personal characteristics (CASPer)-an on-line, video-based screening test-was created. In this paper, we examine the correlation between CASPer and Canadian national licensure examination outcomes in 109 doctors who took CASPer at the time of selection to medical school. Specifically, CASPer scores were correlated against performance on cognitive and 'non-cognitive' subsections of both the Medical Council of Canada Qualifying Examination (MCCQE) Parts I (end of medical school) and Part II (18months into specialty training). Unlike most national licensure exams, MCCQE has specific subcomponents examining personal/professional qualities, providing a unique opportunity for comparison. The results demonstrated moderate predictive validity of CASPer to national licensure outcomes of personal/professional characteristics three to six years after admission to medical school. These types of disattenuated correlations (r=0.3-0.5) are not otherwise predicted by traditional screening measures. These data support the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity, lending evidence of its validation for medical school applicant selection.

Response to: Is it time to rethink the MCCQE Part II?

Response to: Is it time to rethink the MCCQE Part II?

Filling in the gaps of clerkship with a comprehensive clinical skills curriculum

Although the clinical clerkship model is based upon sound pedagogy, including theories of social learning and situated learning, studies evaluating clinical performance of residents suggests that this model may not fully meet the learning needs of students. Here our objective was to design a curriculum to bridge the learning gaps of the existing clerkship model and then evaluate the impact of this on performance on clerkship summative evaluations. We followed Kern's framework to design our curriculum and then compared performance on the clerkship objective structured clinical examination (OSCE), all summative clerkship multiple choice question (MCQ) examinations, and the Medical Council of Canada Qualifying Examination (MCCQE) Part 1 before and after the introduction of our curriculum. In the 2 years following the introduction of our clinical skills curriculum the mean score on the clerkship OSCE was significantly higher than in the 2 years prior to our curriculum [67.12 (5.3) vs. 62.44 (4.93), p < 0.001, d = 0.91]. With the exception of the surgical clerkship MCQ, performance on all clerkship summative MCQ examinations and MCCQE Part 1 was significantly higher following the introduction of our curriculum. In this study we found a significant improvement in the performance on clerks on summative evaluations of knowledge and clinical skills following the introduction of our clinical skills curriculum. Given the unpredictable nature of clinical rotations, the clerkship will always be a risk of failing to deliver the intended curriculum-so medical schools should continue to explore and evaluate ways of changing the delivery of clerkship training to improve learning outcomes.

The Relationship Between Response Time and Diagnostic Accuracy

Psychologists theorize that cognitive reasoning involves two distinct processes: System 1, which is rapid, unconscious, and contextual, and System 2, which is slow, logical, and rational. According to the literature, diagnostic errors arise primarily from System 1 reasoning, and therefore they are associated with rapid diagnosis. This study tested whether accuracy is associated with shorter or longer times to diagnosis. Immediately after the 2010 administration of the Medical Council of Canada Qualifying Examination (MCCQE) Part II at three test centers, the authors recruited participants, who read and diagnosed a series of 25 written cases of varying difficulty. The authors computed accuracy and response time (RT) for each case. Seventy-five Canadian medical graduates (of 95 potential participants) participated. The overall correlation between RT and accuracy was -0.54; accuracy, then, was strongly associated with more rapid RT. This negative relationship with RT held for 23 of 25 cases individually and overall when the authors controlled for participants' knowledge, as judged by their MCCQE Part I and II scores. For 19 of 25 cases, accuracy on each case was positively related to experience with that specific diagnosis. A participant's performance on the test overall was significantly correlated with his or her performance on both the MCCQE Part I and II. These results are inconsistent with clinical reasoning models that presume that System 1 reasoning is necessarily more error prone than System 2. These results suggest instead that rapid diagnosis is accurate and relates to other measures of competence.

A Comparison of Performance Evaluations of Students on Longitudinal Integrated Clerkships and Rotation-Based Clerkships

Longitudinal integrated clerkship (LIC) students typically perform as well as, if not better than, rotation-based clerkship (RBC) students on objective evaluations, yet few studies have compared performance in the clinical setting. This study compared in-training evaluation report (ITER) ratings of LIC and RBC students, including their correlation with more objective evaluations. On the basis of prior academic performance, LIC students (n = 27) at Universities of Alberta, British Columbia, and Calgary were matched with four RBC students from their center. The authors compared reliability of ITER ratings, ITER ratings of clinical skills and professional attributes, and the correlation between ITER ratings and objective evaluations of clinical skills and professional attributes on the Medical Council of Canada Qualifying Examination (MCCQE) Part I. ITER ratings of LIC students were more reliable and significantly higher than those of RBC students for both clinical skills and professional attributes. However, LIC students had lower objective structured clinical examination scores and weaker correlations between subjective and objective evaluations of clinical skills. By comparison, LIC students scored higher on a particular component of the MCCQE and had stronger correlations between subjective and objective evaluations of professionalism. The discrepancy between ratings of LIC students' clinical skills on ITER and other evaluation formats may be due to differences between the content of training and objective evaluations, or systematic rater biases. Further studies are needed to confirm and explain these findings. Promisingly, our data suggest that the LIC model may allow for a more predictive evaluation of professional competencies.

Multiple mini-interviews predict clerkship and licensing examination performance

The Multiple Mini-Interview (MMI) has previously been shown to have a positive correlation with early medical school performance. Data have matured to allow comparison with clerkship evaluations and national licensing examinations. Of 117 applicants to the Michael G DeGroote School of Medicine at McMaster University who had scores on the MMI, traditional non-cognitive measures, and undergraduate grade point average (uGPA), 45 were admitted and followed through clerkship evaluations and Part I of the Medical Council of Canada Qualifying Examination (MCCQE). Clerkship evaluations consisted of clerkship summary ratings, a clerkship objective structured clinical examination (OSCE), and progress test score (a 180-item, multiple-choice test). The MCCQE includes subsections relevant to medical specialties and relevant to broader legal and ethical issues (Population Health and the Considerations of the Legal, Ethical and Organisational Aspects of Medicine[CLEO/PHELO]). In-programme, MMI was the best predictor of OSCE performance, clerkship encounter cards, and clerkship performance ratings. On the MCCQE Part I, MMI significantly predicted CLEO/PHELO scores and clinical decision-making (CDM) scores. None of these assessments were predicted by other non-cognitive admissions measures or uGPA. Only uGPA predicted progress test scores and the MCQ-based specialty-specific subsections of the MCCQE Part I. The MMI complements pre-admission cognitive measures to predict performance outcomes during clerkship and on the Canadian national licensing examination.