Multiple Mini-interview Research Articles

Qualitative analysis of MMI raters' scorings of medical school candidates: A matter of taste?

Recent years have seen leading medical educationalists repeatedly call for a paradigm shift in the way we view, value and use subjectivity in assessment. The argument is that subjective expert raters generally bring desired quality, not just noise, to performance evaluations. While several reviews document the psychometric qualities of the Multiple Mini-Interview (MMI), we currently lack qualitative studies examining what we can learn from MMI raters' subjectivity. The present qualitative study therefore investigates rater subjectivity or taste in MMI selection interview. Taste (Bourdieu 1984) is a practical sense, which makes it possible at a pre-reflective level to apply 'invisible' or 'tacit' categories of perception for distinguishing between good and bad. The study draws on data from explorative in-depth interviews with 12 purposefully selected MMI raters. We find that MMI raters spontaneously applied subjective criteria-their taste-enabling them to assess the candidates' interpersonal attributes and to predict the candidates' potential. In addition, MMI raters seemed to share a taste for certain qualities in the candidates (e.g. reflectivity, resilience, empathy, contact, alikeness, 'the good colleague'); hence, taste may be the result of an ongoing enculturation in medical education and healthcare systems. This study suggests that taste is an inevitable condition in the assessment of students' performance. The MMI set-up should therefore make room for MMI raters' taste and their connoisseurship, i.e. their ability to taste, to improve the quality of their assessment of medical school candidates.

The relationship between Multiple Mini Interview and academic performance of medical students

The relationship between Multiple Mini Interview and academic performance of medical students

If at first you don't succeed… adoption of iPad marking for high-stakes assessments.

Large-scale interview and simulation-based assessments such as objective structured clinical examinations (OSCEs) and multiple mini interviews (MMIs) are logistically complex to administer, generate large volumes of assessment data, and are strong candidates for the adoption of computer-based marking systems. Adoption of new technologies can be challenging, and technical failures, which are relatively commonplace, can delay and/or create resistance to ongoing implementation.This paper reports on the adoption process of an electronic marking system for OSCEs and MMIs following an unsuccessful initial trial. It describes how, after the initial setback, a staged implementation, progressing from small to larger-scale assessments, single to multiple assessment types, and lower to higher stakes assessments, was used to successfully adopt and embed iPad-based marking within our medical school.Critical factors in the success of this approach included thorough appraisal and selection of technologies, rigorous assurance of system reliability and security, constant review and refinement, and careful attention to implementation and end-user training. Engagement of stakeholders is also crucial, especially in the case of previous failures or setbacks. The early identification and recruitment of staff to provide specific expertise and support for adoption of an innovation helps to facilitate this process with four key roles proposed; those of innovation advocate, champion, expert and sponsor.

Reliability and validity of the multiple mini interview (MMI) for admissions to an occupational therapy professional program

Introduction Admissions decisions are among the most consequential assessment points in educating occupational therapists. Suboptimal measures of academic achievement and personal characteristics are often used to select candidates. The purpose of the study is to investigate reliability and validity evidence supporting the use of the multiple mini interview for selecting candidates to an occupational therapy master’s level program. Method The study used a prospective cohort design. Data sources included both archival data and prospective candidate-generated data (multiple mini interview scores, applicant file review scores and academic performance), analyzed using psychometric and correlational approaches. Results Seventy-seven of 80 candidates participated in the prospective component of the study. Overall reliability of a 10-station multiple mini interview was .72. Inter-rater reliability was reasonable. Weak interrelationships were found between multiple mini interview scores and other application components. Conclusion Multiple mini interview scores may be probing different attributes than the other components of the application package. Our study findings support the use of the multiple mini interview for selecting applicants to a professional master’s program in occupational therapy; however, future work should continue to investigate the predictive validity of the multiple mini interview.

First experience with multiple mini interview for medical school admission in Brazil: Does it work in a different cultural scenario?

Purpose: Evaluation of non-cognitive skills never has been used in Brazil. This study aims to evaluate Multiple Mini Interviews (MMI) in the admission process of a School of Medicine in São Paulo, Brazil.Methods: The population of the study comprised 240 applicants summoned for the interviews, and 96 raters. MMI contributed to 25% of the applicants’ final grade. Eight scenarios were created with the aim of evaluating different non-cognitive skills, each one had two raters. At the end of the interviews, the applicants and raters described their impressions about MMI. The reliability of the MMI was analyzed using the Theory of Generalization and Many-Facet Rasch Model (MFRM).Results: The G-study showed that the general reliability of the process was satisfactory (coefficient G = 0.743). The MMI grades were not affected by the raters’ profile, time of interview (p = 0.715), and randomization group (p = 0.353). The Rasch analysis showed that there was no misfitting effects or inconsistent stations or raters. A significant majority of the applicants (98%) and all the raters believed MMIs were important in selecting students with a more adequate profile to study medicine.Conclusions: The general reliability of the selection process was excellent, and it was fully accepted by the applicants and raters.

Utilizing Multiple Mini Interviews in Medical School Admissions

Utilizing Multiple Mini Interviews in Medical School Admissions

The effect of personality traits on undergraduate dental students' performance in multiple mini interviews.

BackgroundMultiple mini interviews (MMI) is a structured student selection method where applicants rotate around a series of stations that do not require previously learned knowledge. The Big Five Inventory (BFI) is a tool often used to measure personality traits commonly used in applicant selection.ObjectiveThe aim of the current study was to assess the effect of personality traits using the BFI on undergraduate dental students’ performance in MMI.MethodsThis research was conducted at The Faculty of Dentistry, King Abdulaziz University, Jeddah, Saudi Arabia. All undergraduates applying to the dental program at King Abdulaziz University, Faculty of Dentistry (KAUFD) in two academic years [(2014–2015) and (2015–2016)] undertook the MMI held at KAUFD and completed the BFI inventory with a total number of 350 students. The MMI consisted of five stations while the BFI contained forty-four items. Data were analysed by SPSS version 20, using independent-samples t-test, Mann-Whitney U test, Spearman’s rho correlation coefficient and Chi-square test.Resultsfemale students scored higher than their male counterparts but scores were only significant in the empathy station (p<0.001) in the first year, and in both empathetic skills and team work abilities (p<0.05) in the second. BFI results indicate that male students scored higher on agreeableness (p<0.003) and lower on neuroticism (p<0.001) in the first year while female students scored higher on agreeableness and conscientiousness (p<0.001) in the second year. Students of private schools had higher total MMI scores compared to those of public schools (p<0.05).ConclusionMMI combined with the Big Five Inventory can be a useful tool in the admission process.

Multiple mini interviews (MMI): A perspective on the first year of MMI at Brighton and Sussex Medical School

Since it was developed at McMaster University School of Medicine in 2002, multiple medical interviews (MMI) have become a popular and accepted method of assessing candidates applying for medical training. Unlike a traditional panel interview it resembles the objective structured clinical examination (OSCE) with numerous 8–10 min stations assessing different topics or scenarios pertinent to medical training and healthcare. This was the first year of implementing MMI at Brighton and Sussex Medical School and my first year of interviewing. I found the process engaging and thought provoking.

Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission

BackgroundThe multiple mini-interview (MMI) is increasingly used for postgraduate medical admissions and in undergraduate settings. MMIs use mostly Situational Questions (SQs) rather than Past-Behavioural Questions (PBQs). A previous study of MMIs in this setting, where PBQs and SQs were asked in the same order, reported that the reliability of PBQs was non-inferior to SQs and that SQs were more acceptable to candidates. The order in which the questions are asked may affect reliability and acceptability of an MMI. This study investigated the reliability of an MMI using both PBQs and SQs, minimising question order bias. Acceptability of PBQs and SQs was also assessed.MethodsForty candidates applying for a postgraduate medical admission for 2016–2017 were included; 24 examiners were used. The MMI consisted of six stations with one examiner per station; a PBQ and a SQ were asked at every station, and the order of questions was alternated between stations. Reliability was analysed for scores obtained for PBQs or SQs separately, and for both questions. A post-MMI survey was used to assess the acceptability of PBQs and SQs.ResultsThe generalisability (G) coefficients for PBQs only, SQs only, and both questions were 0.87, 0.96, and 0.80, respectively. Decision studies suggested that a four-station MMI would also be sufficiently reliable (G-coefficients 0.82 and 0.94 for PBQs and SQs, respectively). In total, 83% of participants were satisfied with the MMI. In terms of face validity, PBQs were more acceptable than SQs for candidates (p = 0.01), but equally acceptable for examiners (88% vs. 83% positive responses for PBQs vs. SQs; p = 0.377). Candidates preferred PBQs to SQs when asked to choose one, though this difference was not significant (p = 0.081); examiners showed a clear preference for PBQs (p = 0.007).ConclusionsReliability and acceptability of six-station MMI were good among 40 postgraduate candidates; modelling suggested that four stations would also be reliable. SQs were more reliable than PBQs. Candidates found PBQs more acceptable than SQs and examiners preferred PBQs when they had to choose between the two. Our findings suggest that it is better to ask both PBQs and SQs during an MMI to maximise acceptability.

Have we got the selection process right? The validity of selection tools for predicting academic performance in the first year of undergraduate medicine.

This article was migrated. The article was marked as recommended. Content: There remains much debate over the 'best' method for selecting students in to medicine. This study aimed to assess the predictive validity of four different selection tools with academic performance outcomes in first-year undergraduate medical students. Methods: Regression analyses were conducted between admission scores on previous academic performance - the Australian Tertiary Admission Rank (ATAR), the Undergraduate Medicine and Health Sciences Admission Test (UMAT), Multiple-Mini Interview (MMI) and the Personal Qualities Assessment (PQA) with student performance in first-year assessments of Multiple Choice Questions, Short Answer Questions, Objective Structured Clinical Examinations (OSCE) and Problem-Based Learning (PBL) Tutor ratings in four cohorts of students (N = 604, 90%). Results: All four selection tools were found to have significant predictive associations with one or more measures of student performance in Year One of undergraduate medicine. UMAT, ATAR and MMI scores consistently predicted first year performance on a number of outcomes. ATAR was the only selection tool to predict the likelihood of making satisfactory progress overall. Conclusions: All four selection tools play a contributing role in predicting academic performance in first year medical students. Further research is required to assess the validity of selection tools in predicting performance in the later years of medicine.

Multiple mini-interview predictive validity for performance on a pharmacy licensing examination.

Predictive validity studies on the use of the multiple mini-interview (MMI) have been primarily in medicine. This study sought to determine the predictive validity of the MMI for performance within a pharmacy programme and on the Pharmacy Examining Board of Canada (PEBC) Qualifying Examination for licensure, and to compare the predictive validity of the MMI with that of pre-pharmacy grade point average (GPA) and Pharmacy College Admission Test (PCAT) score. Admissions data for 223 graduates of the pharmacy programme at the University of Toronto were matched to programme and licensure outcome measures. Multiple linear regression assessed the predictive ability of the MMI, pre-pharmacy GPA, PCAT and covariates for performance in final-year experiential rotations, cumulative GPA (cGPA) and PEBC-MCQ (multiple-choice question examination) and PEBC-OSCE (objective structured clinical examination) overall and subcomponent scores. The PCAT, pre-pharmacy GPA and age significantly predicted the PEBC-MCQ overall score. The MMI was the only significant predictor of overall score on the PEBC-OSCE (β = 0.17, p = 0.02); it also predicted communication and performance subscores. Scores on the PCAT and female gender predicted the communication subscore. Pre-pharmacy GPA, age and female gender significantly predicted cGPA. The MMI was the only significant predictor of institutional/ambulatory rotation score (β = 0.26, p = 0.00). The MMI, designed to measure non-academic attributes including communication, motivation and problem-solving skills, was the only admissions tool with significant predictive validity for performance on the PEBC-OSCE national pharmacy certification examination and in an institutional/ambulatory rotation. These findings, from a single cohort of undergraduates, provide the first report of the predictive validity of the MMI for performance on pharmacy licensure examinations and thereby strengthen the evidence for its use in health professions selection. Prior university academic performance significantly predicted cGPA and performance on the PEBC-MCQ. Performance on the PCAT also predicted PEBC-MCQ results.

Multiple Mini-Interviews in the Age of the Internet: Does Preparation Help Applicants to Medical School?

Many medical schools have adopted multiple mini-interviews (MMI) as an advanced selection tool. MMIs are expensive and used to test only a few dozen candidates per day, making it infeasible to develop a different test version for each test administration. Therefore, some items are reused both within and across years. This study investigated the influence of item reuse and familiarity with test structure on scores for a high-stakes non-cognitive test administered every year to hundreds of medical school applicants. The results show no evidence that reusing items makes the test easier or less differentiating. Data are consistent across years, test versions and item types. Two additional aspects of familiarity with test structure and test items were examined: whether retaking the test has an effect and how candidates prepare for the test. It was found that retaking the test in and of itself does not result in higher scores. Notwithstanding the results, in the age of the Internet, test developers must assume that all items included in any particular test version will be fully exposed. Hence, in order to preserve test validity, reliability and fairness, it is important to rethink and carefully preplan test versions, as well as decide what information to divulge to candidates.

Multiple mini interviews: view from medical student interviewers.

Multiple mini interviews: view from medical student interviewers.

Multiple Mini-Interview for Postgraduate Year Residency Recruitment: First Experience in Taiwan

Background: We investigated the effectiveness of the first hospital-held multiple mini-interview for recruitment into a postgraduate residency program in Taiwan. Methods: We conducted a multiple mini-interview to recruit 37 trainees into the postgraduate general medicine residency in January 2017 of the 37 postgraduate year candidates, 28 had interned at our hospital. Between-group correlations among subgroup analyses using multiple comparisons, Chi square test, and the generalized linear model were conducted to test the validity and discrimination of the multiple mini-interview questions. Correlations between multiple mini-interview scores, objective structured clinical examination scores, and clinical practice examinations were conducted and available for 19 applicants who interned at this hospital to test the effectiveness of the multiple mini-interview as a recruiting tool. Results: The multiple mini-interview process exhibited high test discrimination (P<0.001). Whether an intern studied at our hospital or at a different hospital did not affect the performance of the multiple mini-interview. Significant correlations were observed among performance in the multiple mini-interview, all three objective structured clinical examinations, and in four of six clinical practice departments among the 19 candidates who interned at our hospital (P<0.05). Conclusions: The significant correlation of the multiple mini-interviews with clinical performance scores provides powerful support for multiple mini-interview assessment in recruiting residents suitable for clinical work. Continuing to collect data to study the multiple mini-interview program would improve our understanding of it and provide better support for the postgraduate year recruitment program.

The predictive validity of using admissions testing and multiple mini-interviews in undergraduate university admissions

ABSTRACTThe aim of this study was to evaluate the predictive validity of a two-step admissions procedure that included a cognitive ability test followed by multiple mini-interviews (MMIs) used to assess non-cognitive skills, compared to grade-based admissions relative to subsequent drop-out rates and academic achievement after one and two years of study. The participants consisted of the entire population of 422 psychology students who were admitted to the University of Southern Denmark between 2010 and 2013. The results showed significantly lower drop-out rates after the first year of study, and non-significant lower drop-out rates after the second year of study for the admission procedure that included the assessment of non-cognitive skills though the MMI and the admissions test. Furthermore, this admission procedure resulted in a significant lower risk of failing the final exam after the first and second year of study, compared to the grade-based admissions procedure. Finally, students admitted through the MMI and the admissions test scored significantly higher on academic learning self-efficacy and critical thinking compared to students selected by grades. The implications for higher education admissions are discussed.

Impact of sociodemographic characteristics of applicants in multiple mini-interviews

Background: Multiple mini-interviews (MMI) are commonly used for medical school admission. This study aimed to assess if sociodemographic characteristics are associated with MMI performance, and how they may act as barriers or enablers to communication in MMI.Methods: This mixed-method study combined data from a sociodemographic questionnaire, MMI scores, semi-structured interviews and focus groups with applicants and assessors. Quantitative and qualitative data were analyzed using multiple linear regression and a thematic framework analysis.Results: 1099 applicants responded to the questionnaire. A regression model (R2 = 0.086) demonstrated that being age 25–29 (β = 0.11, p = 0.001), female and a French-speaker (β = 0.22, p = 0.003) were associated with better MMI scores. Having an Asian-born parent was associated with a lower score (β = −0.12, p < 0.001). Candidates reporting a higher family income had higher MMI scores. In the qualitative data, participants discussed how maturity and financial support improved life experiences, how language could act as a barrier, and how ethnocultural differences could lead to misunderstandings.Conclusion: Age, gender, ethnicity, socioeconomic status and language seem to be associated with applicants’ MMI scores because of perceived differences in communications skills and life experiences. Monitoring this association may provide guidance to improve fairness of MMI stations.

Identifying applicants suitable to a career in nursing: a value-based approach to undergraduate selection.

The aim of this study was to complement existing evidence on the suitability of Multiple Mini Interviews as a potential tool for the selection of nursing candidates on to a BSc (Hons) nursing programme. This study aimed to trial the Multiple Mini Interview approach to recruitment with a group of first year nursing students (already selected using traditional interviews). Cross-sectional validation study. This paper reports on the evaluation of the participants' detailed scores from the Multiple Mini Interview stations; their original interview scores and their end of year results. This study took place in March 2015. Scores from the seven Multiple Mini Interview stations were analysed to show the internal structure, reliability and generalizability of the stations. Original selection scores from interviews and in-course assessment were correlated with the MMI scores and variation by students' age, gender and disability status was explored. Reliability of the Multiple Mini Interview score was moderate (G=0·52). The Multiple Mini Interview score provided better differentiation between more able students than did the original interview score but neither score was correlated with the module results. Multiple Mini Interview scores were positively associated with students' age but not their gender or disability status. The Multiple Mini Interview reported in this study offers a selection process that is based on the values and personal attributes regarded as desirable for a career in nursing and does not necessarily predict academic success. Its moderate reliability indicates the need for further improvement but it is capable of discriminating between candidates and shows little evidence of bias.

Twelve Tips on Implementing Multiple Mini-Interviews in a Hybrid Admissions Model

This article was migrated. The article was not marked as recommended. Background: Multiple mini-interviews (MMI) were proposed almost a decade ago as an alternate means of assessing medical school applicants. MMIs are similar to an OSCE (Objectively Structured Clinical Examination) in that applicants complete a series of brief encounters focused on non-clinical topics. Aim: To assist those considering the addition of MMIs with standard interviews in a hybrid admissions model. Methods: The tips are derived from personal experience with a hybrid model at one academic institution. Results: The twelve tips describe approaches for obtaining early faculty buy-in, designing a floor plan and time-table, aligning MMI content with the program mission and goals, conducting faculty training, developing contingency plans, managing applicants' expectations, encouraging committee participation, promoting holistic review of each applicant, utilizing the benefits the model allows, changing scenarios periodically, studying the model, and sharing the lessons learned. Conclusion: It is hoped these suggestions are helpful to those considering a hybrid model for their institution.

Breaking bad news in a multiple mini interview: advice from an admissions dean.

Breaking bad news in a multiple mini interview: advice from an admissions dean.

Can Multiple Mini‐Interviews Predict Academic Performance of Dental Students? A Two‐Year Follow‐Up

Prior research has shown that students' previous grade point average (GPA) is the best predictor for future academic success. However, it can only partly predict the variability in dental school performance. The aim of this study was to assess the predictive value of multiple mini-interviews (MMI) as an admission criterion by comparing them with the academic performance of dental students over a two-year period. All incoming undergraduate dental students at the King Abdulaziz University Faculty of Dentistry (KAUFD) during academic year 2013-14 were invited to participate in MMI. Students rotated through six objective structured clinical exam (OSCE)-like stations for 30 minutes total and were interviewed by two trained faculty interviewers at each station. The stations were focused on noncognitive skills thought to be essential to academic performance at KAUFD. The academic performance of these students was then followed for two years and linked to their MMI scores. A total of 146 students (71 males and 75 females) participated in an interview (response rate=92.9%). Most students scored in the acceptable range at each MMI station. Students' total MMI score, ambitions, and motives were significant predictors of GPA during the two years of follow-up (p<0.038 and p<0.001, respectively). In this study, MMI was found to be able to predict future academic performance of undergraduate dental students.