Multiple Mini-interview Process Research Articles

Feasibility and Acceptability of a Multiple Mini-interview Format for Evaluation of Small Animal Internal Medicine Residency and Internship Candidates.

The multiple mini-interview (MMI) format assesses candidates' performance in various competencies and is becoming commonplace in medical school and residency programs. This interview format compares to and surpasses the traditional interview in validity, reliability, feasibility, and acceptability. We developed a MMI to assess resident and specialty intern candidates for Small Animal Internal Medicine over a 3-year period. Our aims were to assess acceptability by obtaining impressions from interviewers and candidates of the MMI process and to evaluate feasibility by quantifying time required. In total, 61 resident candidates completed the survey, a response rate of 70% of total interviewees. Respondents reported the MMI as more stressful (82%, 95% CI: 70%, 91%) and less enjoyable (62%, 95% CI: 49%, 74%). While 54% (95% CI: 41%, 67%) of respondents preferred a traditional interview process, 70% (95% CI: 57%, 81%) perceived that the MMI was fairer and 51% (95% CI: 38%, 64%) felt it allowed them to better demonstrate their strengths. Most (63%, 95% CI: 49%, 75%) reported that their experience with the MMI would lead them to rank the program more highly. Interviewers preferred the MMI process due to improved perceived fairness and time efficiency. MMI required less total time, 29.5 hours versus 112 hours for the traditional interview over 3-years (p = .002) and lower per applicant time investment (.34 hours/applicant vs. 1.2 hours/applicant, p = .0001). In conclusion, the MMI process was acceptable to interviewees, preferred by interviewers, and was feasible in terms of time savings compared to traditional interviews.

Experiences of Rural and Metropolitan Background Applicants in Preparing for and Completing a Regionally Focused Multiple Mini-interview for Admission into a Regional Medical Program.

To better target rural background and rurally interested applicants during medical school admission, it is increasingly common for rural medical programs to include multiple mini-interview (MMI) scenarios designed to screen for rural interest. It remains unclear whether the inclusion of regionally/rurally focused MMI scenarios positively impacts the selection of rural background applicants and evidence is limited regarding why rural background applicants may perform worse on the MMI. Therefore, this study explored how rural and metropolitan applicants prepare for and perceive the MMI for admission to a regional medical pathway. A mixed-methods survey was sent to provisional entry regional pathway medical school applicants who had completed an MMI. The survey was distributed before any offers of admission had been released. Rural applicants spent less time and money preparing for the MMI and felt less prepared (P < 0.05). However, time and money spent, and resources used to prepare were not associated with feeling more prepared (all P > 0.05). Respondents mostly felt that the MMI process aligned with their expectations (83%), is fair (64%), and helps a rural program select the most suitable applicants (61%). Rural applicants generally felt that they had an advantage over other applicants (61%) while most metropolitan applicants did not (23%; P = 0.002). Applicants to a regional medical pathway are generally supportive of the MMI process. It appears valuable for applicants to prepare for the MMI by understanding the format and requirements; however, investing substantial time and money does not underpin feeling better prepared. MMI scenarios which include a regional focus are perceived to advantage rural applicants.

Qualitative Analysis of Multiple Mini Interview Interviewer Comments.

Qualitative studies of the Multiple Mini Interview (MMI) have investigated the attitudes and thoughts of prospective students and interviewers (i.e., raters) on the MMI interview, but none have examined rater's written assessments. Concerns regarding what the MMI measures, especially across and within each interview, have sparked investigations to determine how and what raters are measuring. Raters communicate their student evaluation(s) through numerical ratings and written comments that provide score context. This study explores rater's written comments to better understand the specific information gathered during the MMI process that contributes to interviewee evaluations. Randomized data from two US medical schools were examined with no numerical scores or other information about the interviewee provided to reviewers. In reviewing the rater comments, common words and phrases were identified to help construct themes that characterized the content (domains). Authors reviewed each other's notes and comments regarding themes and worked together to verify themes for accuracy. Using a directed content approach to content analysis and reviewing the rater's comments, the results indicate that raters are focused on seven different domains: perspective taking, presentation, qualities, communication, coherence, comprehension, and non-verbal. Many of the rater comments contained multiple themes. Raters' MMI comments provide the context for numerical scores allowing admissions committees to more fully understand a candidate's strengths or weaknesses. Identifying the themes in rater comments can ultimately assist the admissions committee to more comprehensively understand assessment elements that raters are using and consider important during the MMI evaluation.

Multiple Mini-interview Consistency and Satisfactoriness for Residency Program Recruitment: Oman Evidence.

ObjectivesStandard interviews are used by most residency programs to assess non-cognitive skills, but variability in the interviewer’s skills, interviewer bias, and context specificity limit reliability. We sought to investigate the consistency and satisfactoriness of the multiple mini-interview (MMI) model for resident selection into an otorhinolaryngology head and neck surgery residency program.MethodsThis pilot study was done in an independent academic residency training center for 15 applicants, in seven eight-minute MMI stations with eight raters for the 2015–2016 academic year. The raters included the chief resident and education committee chairman in one of the stations. Candidates were assessed on two items: medical knowledge (two standardized case scenarios) and behavioral knowledge (personality and attitude, professionalism, communication, enthusiasm to the specialty, and English proficiency).ResultsOf 15 candidates, 10 (66.7%) were female and five (33.3%) were male; five were recommended for selection, and five were kept on the waiting list. The reliability, intraclass correlation coefficient (ICC), of the scores obtained from seven items of MMI was 0.36 (95% confidence interval (CI): -0.31–0.75; p = 0.110). However, the ICC of the medical interview was 0.54 (95% CI: 0.45–0.84; p = 0.090). The correlation between behavioral items score and MMI total score was r = 0.135 (p = 0.150).ConclusionsThe interview evaluation/survey form given to candidates and interviewers has shown that MMI is a fair and effective tool to evaluate non-cognitive traits. Both candidates and interviewers prefer MMI to standard interviews. The MMI process for residency interviews can generate reliable interview results using only seven stations and is acceptable and preferred over standard interview modalities by residency program applicants and faculty members.

Using multiple mini interviews as a pre-screening tool for medical student candidates completing international health electives

ABSTRACTThere continues to be an increase in the number of learners who participate in international health electives (IHEs). However, not all learners enter IHEs with the same level of knowledge, attitude, and previous experience, which puts undue burden on host supervisors and poses risks to student and patient safety. The Multiple Mini-Interview (MMI) is a technique that has become a popular method for undergraduate and postgraduate-level health science admissions programs. This paper describes the MMI process used by our program to screen first-year medical students applying for pre-clinical IHEs. Two country-specific cases were developed to assess non-cognitive skills. One hundred percent (100%) of the students (n = 48) and interviewers (n = 10) who participated in MMIs completed anonymous surveys on their experience. The majority of students rated the scenarios as realistic (>90%); 96% found the MMI format fair and balanced; 96% of students felt that they were able to clearly articulate their thoughts; 75% of students stated that they had a general understanding of how the MMIs worked; only 33% of students would have preferred a traditional one-to-one interview. Feedback from both interviewers and students was positive toward the MMI experience, and no students were identified as unfit for participation. Ultimately, 43 students participated in pre-clinical IHEs in 2016. In this paper, we will outline our MMI process, detail shortcomings, and discuss our next steps to screen medical students for IHEs.

Multiple Mini-Interview for Postgraduate Year Residency Recruitment: First Experience in Taiwan

Background: We investigated the effectiveness of the first hospital-held multiple mini-interview for recruitment into a postgraduate residency program in Taiwan. Methods: We conducted a multiple mini-interview to recruit 37 trainees into the postgraduate general medicine residency in January 2017 of the 37 postgraduate year candidates, 28 had interned at our hospital. Between-group correlations among subgroup analyses using multiple comparisons, Chi square test, and the generalized linear model were conducted to test the validity and discrimination of the multiple mini-interview questions. Correlations between multiple mini-interview scores, objective structured clinical examination scores, and clinical practice examinations were conducted and available for 19 applicants who interned at this hospital to test the effectiveness of the multiple mini-interview as a recruiting tool. Results: The multiple mini-interview process exhibited high test discrimination (P<0.001). Whether an intern studied at our hospital or at a different hospital did not affect the performance of the multiple mini-interview. Significant correlations were observed among performance in the multiple mini-interview, all three objective structured clinical examinations, and in four of six clinical practice departments among the 19 candidates who interned at our hospital (P<0.05). Conclusions: The significant correlation of the multiple mini-interviews with clinical performance scores provides powerful support for multiple mini-interview assessment in recruiting residents suitable for clinical work. Continuing to collect data to study the multiple mini-interview program would improve our understanding of it and provide better support for the postgraduate year recruitment program.

Perceptions and Cost-Analysis of a Multiple Mini-Interview in a Pharmacy School Admissions Process

To improve the quality of admissions interviews for a doctor of pharmacy program, using a multiple mini-interview (MMI) in place of the standard interview. Stakeholders completed an anonymous web-based survey. This study characterized perceptions of the MMI format across 3 major stakeholders (candidates, interviewers, admissions committee members) and included comparative cost estimates.Costs were estimated using human and facility resources from the 2012 cycle (standard format) and the 2013 cycle (MMI format). Most candidates (65%), interviewers (86%), and admissions committee members (79%) perceived the MMI format as effective for evaluating applicants, and most (59% of candidates, 84% of interviewers, 77% of committee members) agreed that the MMI format should be continued. Cost per candidate interviewed was $136.34 (standard interview) vs $75.30 (MMI). Perceptions of the MMI process were favorable across stakeholder groups, and this format was less costly per candidate interviewed.

Rater Perceptions of Bias Using the Multiple Mini-interview Format: a Qualitative Study

Introduction The Multiple Mini-Interview (MMI) format appears to mitigate individual rater biases. However, the format itself may introduce structural systematic bias, favoring extroverted personality types. This study aimed to gain a better understanding of these biases from the perspective of the interviewer. Methods A sample of MMI interviewers participated in a series of primary and follow-up one-on-one semi-structured interviews. Interviews pursued subjects of perception of biases (including norming; applicant personality, appearance and behavior; and interviewer personality) associated with the MMI process. Emergent qualitative data analysis was performed using the constant-comparative method. Results A number of perceived biases were identified by subjects, sub-grouped into cultural factors, personality factors, perception of prior preparation, concerns with norming, and biases associated with specific applicant characteristics. Discussion While the MMI appears to help mitigate individual rater biases, our analysis suggests that raters perceive structural systematic biases may be introduced by the question type and format of the MMI itself. Whether rater awareness of these biases mitigates them, and whether these herald other unconscious biases is unknown.

Prognostic Value of the MULTIPLE MINI-INTERVIEW for Emergency Medicine Residency Performance

BackgroundThe Multiple Mini-Interview (MMI) uses short, structured contacts, and is known to predict medical school success better than traditional interviews and application materials. Its utility in Emergency Medicine residency selection is untested. ObjectivesWe investigate whether it provides additional information regarding future first-year resident performance that can be useful in resident selection. MethodsFrom three Emergency Medicine residency programs, 71 interns in their first month completed an MMI developed to focus on desirable resident characteristics. Application data were reviewed. First-year resident performance assessments covering the American Council for Graduate Medical Education (ACGME) core competencies, along with professionalism and performance concerns, were obtained. Multiple logistic regressions were employed and MMI correlations were compared with program rank lists and typical selection factors. ResultsAn individual's score on the MMI correlated with overall performance (p < 0.05) in single logistic regression. MMI correlated with ACGME individual competencies patient care and procedural skills at a less robust level (p < 0.1), but not with any other outcomes. Rank list position correlated with the diagnostic skill competency (p < 0.05), but no others. Traditional selection factors correlated with overall performance, disciplinary action, patient care, medical knowledge, and diagnostic skills (p < 0.05). MMI was not correlated significantly with the outcomes when included in multiple ordinal logistic regression with other selection factors. ConclusionsMMI scores correlate with overall performance, but are not statistically significant when other traditional selection factors were considered. The MMI process seems potentially superior to program rank list at correlating with first-year performance. The MMI may provide additional benefit when examined using a larger and more diverse sample.

Validating MMI scores: are we measuring multiple attributes?

The multiple mini-interview (MMI) used in health professional schools' admission processes is reported to assess multiple non-cognitive constructs such as ethical reasoning, oral communication, or problem evaluation. Though validation studies have been performed with total MMI scores, there is a paucity of information regarding how well MMI scores differentiate the constructs being measured, the relationship between MMI scores (construct or total) and personality characteristics, and how well MMI scores (construct or total) predict future performance in practice. Results from these studies could assist with MMI station development, rater training, score interpretation, and resource allocation. The purpose of this study was to investigate the validity of MMI construct scores (oral communication and problem evaluation), and their relationship to personality measures (emotionality and extraversion) and specific scores from standardized clinical communications interviews (building the relationship and explaining and planning). Confirmatory factor analysis results support a two factor MMI model, however the correlation between these factors was .87. Oral communication MMI scores significantly correlated with extraversion (r c = .25, p < .05), but MMI scores were not related to emotionality. Scores for building a relationship were significantly related to MMI oral communication scores, (r c = .46, p < .001) and problem evaluation scores (r c = .43, p < .001); scores for explaining and planning were significantly related to MMI problem evaluation scores (r c = .36, p < .01). The results provide validity evidence for assessing multiple non-cognitive attributes during the MMI process and reinforce the importance of developing MMI stations and scoring rubrics for attributes identified as important for future success in school and practice.

Applicants' perceptions on the multiple mini-interview process as a selection tool for dental and therapy and hygiene students

This research details and quantifies applicants' perceptions towards multiple mini-interviews as a selection tool. BDS and DCP (therapy and hygiene) candidates who secured a multiple mini-interview (MMI) completed a questionnaire based on the thematic analysis of the previous year's multiple mini-interviews. The questionnaire explored the candidates' feelings with reference to the interview process and the stations used on the MMI process. Of 235 BDS and 62 therapy and hygiene interviewees, 231 (98.3%) and 61 (96.7%) completed or partially completed the questionnaire. Demographic data revealed that 61.9% (143) of the BDS interviewees were female and 86.7% were female (52) in the therapy and hygiene cohort. The majority of interviewees came from schools in England and received a state education. While 69.6% of the BDS interviewees had experience of an interview before their MMI, this was down to 58.3% in the hygiene and therapy interviewees. Binomial statistical calculations and chi-squared tests of independence of categorical variables on nominally scaled data revealed statistically significant differences (p >0.001) and both groups of students were positive regarding the selection process (p >0.001). Overall there was similarity between groups with respect to the themes evaluated and stations used in the MMI process, however, statistical analysis did highlight areas of difference. There were 54 separate free text comments recorded but qualitative analysis failed to identify any themes that had not been previously identified. This research did not reveal any new emergent themes in relation to the MMI process between two different cohorts of students but did reveal general similarities and some specific areas of difference.

Utility of a Writing Station in the Multiple Mini-Interview

The multiple mini-interview (MMI) is a reliable and valid method of selecting applicants for admission to health professional schools on the basis of non-cognitive traits. Because the MMI is a series of short interview stations that applicants rotate through in coordinated sequence, it can potentially be resource intensive. However, the MMI design has room for innovation and efficiency. At the University of Manitoba Faculty of Medicine, a 10-minute unsupervised writing station (WS) was incorporated into the MMI to obtain a writing sample from each applicant, to increase the number of independent scores per applicant, and to increase the number of applicants interviewed per circuit without increasing interviewer numbers. One assessor evaluated all the writing samples and assigned a score ranging from 1 to 7. With the inclusion of a WS into an 11-station MMI, the faculty's capacity to interview applicants increased by 9% (from 297 to 324) without substantially increasing interviewer hours needed per day. For 1,257 applicants interviewed in 2008-2011, the mean WS score was 4.03 (SD=1.36), whereas applicants' mean of 10 oral station (OS) scores was 4.62 (SD=0.69). Correlations between WS score and mean OS score ranged from .16 to .27 (p<.01) over the four years. Because inter-station correlations for OS ranged from .01 to .37, the correlation of .21 between WS and mean OS scores for all four years combined appears reasonable. Institutions that want to effectively increase the capacity of their MMI process might consider adding a WS.

Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses

The multiple mini-interview (MMI) has become an increasingly popular admissions method for selecting prospective students into professional programs (e.g., medical school). The MMI uses a series of short, labour intensive simulation stations and scenario interviews to more effectively assess applicants' non-cognitive qualities such as empathy, critical thinking, integrity, and communication. MMI data from 455 medical school applicants were analyzed using: (1) Generalizability Theory to estimate the generalizability of the MMI and identify sources of error; and (2) the Many-Facet Rasch Model, to identify misfitting examinees, items and raters. Consistent with previous research, our results support the reliability of MMI process. However, it appears that the non-cognitive qualities are not being measured as unique constructs across stations.

Reliability and acceptability of a five-station multiple mini-interview model for residency program recruitment

BackgroundStandard interviews are used by most residency programs in the United States for assessment of aptitude of the non-cognitive competencies, but variability of interviewer skill, interviewer bias, interviewer leniency or stringency, and context specificity limit reliability.AimTo investigate reliability and acceptability of five-station multiple mini-interview (MMI) model for resident selection into an internal medicine residency program in the United States.SettingOne independent academic medical center.ParticipantsTwo hundred and thirty-seven applicants and 17 faculty interviewers.Program descriptionFive, 10-min MMI stations with five different interviewers blinded to the candidate's records and one traditional 20-min interview with the program director. Candidates were rated on two items: interpersonal and communication skills, and overall performance.Program evaluationGeneralizability data showed that the reliability of our process was high (>0.9). The results of anonymous surveys demonstrated that both applicants and interviewers consider the MMI as a fair and more effective tool to evaluate non-cognitive traits, and prefer the MMI to standard interviews.DiscussionThe MMI process for residency interviews can generate reliable interview results using only five stations, and it is acceptable and preferred over standard interview modalities by the applicants and faculty members of one US residency program.

Association Between a Medical School Admission Process Using the Multiple Mini-interview and National Licensing Examination Scores

There has been difficulty designing medical school admissions processes that provide valid measurement of candidates' nonacademic qualities. To determine whether students deemed acceptable through a revised admissions protocol using a 12-station multiple mini-interview (MMI) outperform others on the 2 parts of the Canadian national licensing examinations (Medical Council of Canada Qualifying Examination [MCCQE]). The MMI process requires candidates to rotate through brief sequential interviews with structured tasks and independent assessment within each interview. Cohort study comparing potential medical students who were interviewed at McMaster University using an MMI in 2004 or 2005 and accepted (whether or not they matriculated at McMaster) with those who were interviewed and rejected but gained entry elsewhere. The computer-based MCCQE part I (aimed at assessing medical knowledge and clinical decision making) can be taken on graduation from medical school; MCCQE part II (involving simulated patient interactions testing various aspects of practice) is based on the objective structured clinical examination and typically completed 16 months into postgraduate training. Interviews were granted to 1071 candidates, and those who gained entry could feasibly complete both parts of their licensure examination between May 2007 and March 2011. Scores could be matched on the examinations for 751 (part I) and 623 (part II) interviewees. Admissions decisions were made by combining z score transformations of scores assigned to autobiographical essays, grade point average, and MMI performance. Academic and nonacademic measures contributed equally to the final ranking. Scores on MCCQE part I (standardized cut-score, 390 [SD, 100]) and part II (standardized mean, 500 [SD, 100]). Candidates accepted by the admissions process had higher scores than those who were rejected for part I (mean total score, 531 [95% CI, 524-537] vs 515 [95% CI, 507-522]; P = .003) and for part II (mean total score, 563 [95% CI, 556-570] vs 544 [95% CI, 534-554]; P = .007). Among the accepted group, those who matriculated at McMaster did not outperform those who matriculated elsewhere for part I (mean total score, 524 [95% CI, 515-533] vs 546 [95% CI, 535-557]; P = .004) and for part II (mean total score, 557 [95% CI, 548-566] vs 582 [95% CI, 569-594]; P = .003). Compared with students who were rejected by an admission process that used MMI assessment, students who were accepted scored higher on Canadian national licensing examinations.

Enhancing the Reliability of the Multiple Mini-Interview for Selecting Prospective Health Care Leaders

The David Geffen School of Medicine at UCLA Program in Medical Education (UCLA-PRIME) used a 12-station multiple mini-interview (MMI) circuit to assess applicants. The authors sought to determine the reliability of the MMI, potential bias in scores, and the degree of acceptance by interviewers and applicants. In 2009, 28 interviewers interviewed a cohort of 76 applicants. An anonymous survey assessed interviewers' and applicants' satisfaction with the MMI process and perceived bias. Psychometric properties were determined with generalizability and decision theory. The process was repeated the following year with a new cohort of 78 applicants and minor modifications aimed at improving reliability. The MMI format was well received by both applicants and interviewers. No bias based on gender or disadvantaged status was found. The preliminary reliability of the MMI in 2009 was 0.58-lower than reported in previous studies-but improved in 2010 to 0.71 after an easy station was replaced with a more challenging one and a new scoring rubric was introduced. This interview technique proved to be reliable and was seen as transparent, uniform, and fair. The predictive validity of this process remains to be determined.