Multiple Mini-interview Format Research Articles

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format.

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors. We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework. Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment. While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

Feasibility and Acceptability of a Multiple Mini-interview Format for Evaluation of Small Animal Internal Medicine Residency and Internship Candidates

The multiple mini-interview (MMI) format assesses candidates’ performance in various competencies and is becoming commonplace in medical school and residency programs. This interview format compares to and surpasses the traditional interview in validity, reliability, feasibility, and acceptability. We developed a MMI to assess resident and specialty intern candidates for Small Animal Internal Medicine over a 3-year period. Our aims were to assess acceptability by obtaining impressions from interviewers and candidates of the MMI process and to evaluate feasibility by quantifying time required. In total, 61 resident candidates completed the survey, a response rate of 70% of total interviewees. Respondents reported the MMI as more stressful (82%, 95% CI: 70%, 91%) and less enjoyable (62%, 95% CI: 49%, 74%). While 54% (95% CI: 41%, 67%) of respondents preferred a traditional interview process, 70% (95% CI: 57%, 81%) perceived that the MMI was fairer and 51% (95% CI: 38%, 64%) felt it allowed them to better demonstrate their strengths. Most (63%, 95% CI: 49%, 75%) reported that their experience with the MMI would lead them to rank the program more highly. Interviewers preferred the MMI process due to improved perceived fairness and time efficiency. MMI required less total time, 29.5 hours versus 112 hours for the traditional interview over 3-years ( p = .002) and lower per applicant time investment (.34 hours/applicant vs. 1.2 hours/applicant, p = .0001). In conclusion, the MMI process was acceptable to interviewees, preferred by interviewers, and was feasible in terms of time savings compared to traditional interviews.

Best Practices for Reducing Bias in the Interview Process.

Purpose of ReviewObjective measures of residency applicants do not correlate to success within residency. While industry and business utilize standardized interviews with blinding and structured questions, residency programs have yet to uniformly incorporate these techniques. This review focuses on an in-depth evaluation of these practices and how they impact interview formatting and resident selection.Recent FindingsStructured interviews use standardized questions that are behaviorally or situationally anchored. This requires careful creation of a scoring rubric and interviewer training, ultimately leading to improved interrater agreements and biases as compared to traditional interviews. Blinded interviews eliminate even further biases, such as halo, horn, and affinity bias. This has also been seen in using multiple interviewers, such as in the multiple mini-interview format, which also contributes to increased diversity in programs. These structured formats can be adopted to the virtual interviews as well.SummaryThere is growing literature that using structured interviews reduces bias, increases diversity, and recruits successful residents. Further research to measure the extent of incorporating this method into residency interviews will be needed in the future.

Multiple mini-interview for undergraduate allied health programs in Singapore

Aims: To investigate the acceptability and feasibility of a four-station multiple mini-interview (MMI) model for admission selection into the four allied health undergraduate programs in Singapore. Methods: From 2015 to 2017, applicants to the four allied health undergraduate programs (physical therapy, occupational therapy, diagnostic radiography, and radiation therapy) at a university in Singapore underwent a four-station MMI. In 2017, interviewers of the MMI were polled using a 7-point Likert scale survey. Descriptive statistics were performed to analyze the quantitative portion of the survey, while content analysis and thematic description were applied to the qualitative data. Results: Data was collected from 79 interviewers. The majority agreed that the MMI was a fair process for applicants (83.5%), a good way of assessing interest and potential to become healthcare professionals (78.5%), and tested more aspects of the applicants than the traditional interviews (72%). They concluded that the MMI format was structured, objective, rigorous, unbiased, and assessed a valid range of competencies (67%). Conclusion: This study assessed and established the acceptability and feasibility of the MMI for undergraduate admissions to the allied health programs in Singapore.

Transition to multiple mini interview (MMI) interviewing for medical school admissions.

IntroductionThe multiple mini interview (MMI) has been incorporated into the holistic review process in the selection of students to US medical schools. The MMI has been used to evaluate interpersonal and intrapersonal attributes which are deemed as necessary for future physicians. We hypothesized that there would be little difference in overall MMI evaluation data compared with traditional interview ratings.MethodsThe University of North Carolina School of Medicine developed an interview process that included a traditional interview and MMI format during the 2019 admissions cycle. Evaluation data along with key demographic variables for 608 MD program applicants were analyzed using descriptive and inferential statistical analyses.ResultsThe MMI format slightly favored female over male applicants (p = 0.002) but did not select for or against applicants based on age, race/ethnicity, underserved/rural area upbringing, or indicators of disadvantage. Out of 608 applicants, 356 (59%) completed a post-interview survey in which the experience was positively rated.DiscussionBased on our experience, the use of a hybrid model of traditional interviews complemented with MMI stations provided greater details in the assessment of medical school applicants while obtaining equivalent data and acceptability amongst applicants.

Using multiple mini interviews as a pre-screening tool for medical student candidates completing international health electives

ABSTRACTThere continues to be an increase in the number of learners who participate in international health electives (IHEs). However, not all learners enter IHEs with the same level of knowledge, attitude, and previous experience, which puts undue burden on host supervisors and poses risks to student and patient safety. The Multiple Mini-Interview (MMI) is a technique that has become a popular method for undergraduate and postgraduate-level health science admissions programs. This paper describes the MMI process used by our program to screen first-year medical students applying for pre-clinical IHEs. Two country-specific cases were developed to assess non-cognitive skills. One hundred percent (100%) of the students (n = 48) and interviewers (n = 10) who participated in MMIs completed anonymous surveys on their experience. The majority of students rated the scenarios as realistic (>90%); 96% found the MMI format fair and balanced; 96% of students felt that they were able to clearly articulate their thoughts; 75% of students stated that they had a general understanding of how the MMIs worked; only 33% of students would have preferred a traditional one-to-one interview. Feedback from both interviewers and students was positive toward the MMI experience, and no students were identified as unfit for participation. Ultimately, 43 students participated in pre-clinical IHEs in 2016. In this paper, we will outline our MMI process, detail shortcomings, and discuss our next steps to screen medical students for IHEs.

How many schools adopt interviews during the student admission process across the health professions in the United States of America?

Health profession schools use interviews during the admissions process to identify certain non-cognitive skills that are needed for success in diverse, inter-professional settings. This study aimed to assess the use of interviews during the student admissions process across health disciplines at schools in the United States of America in 2014. The type and frequency of non-cognitive skills assessed were also evaluated. Descriptive methods were used to analyze a sample of interview rubrics collected as part of a national survey on admissions in the health professions, which surveyed 228 schools of medicine, dentistry, pharmacy, nursing, and public health. Of the 228 schools, 130 used interviews. The most desirable non-cognitive skills from 34 schools were identified as follows: communication skills (30), motivation (22), readiness for the profession (17), service (12), and problem-solving (12). Ten schools reported using the multiple mini-interview format, which may indicate potential for expanding this practice. Disparities in the use of interviewing across health professions should be verified to help schools adopt interviews during student admissions processes.

Perceptions and Cost-Analysis of a Multiple Mini-Interview in a Pharmacy School Admissions Process

To improve the quality of admissions interviews for a doctor of pharmacy program, using a multiple mini-interview (MMI) in place of the standard interview. Stakeholders completed an anonymous web-based survey. This study characterized perceptions of the MMI format across 3 major stakeholders (candidates, interviewers, admissions committee members) and included comparative cost estimates.Costs were estimated using human and facility resources from the 2012 cycle (standard format) and the 2013 cycle (MMI format). Most candidates (65%), interviewers (86%), and admissions committee members (79%) perceived the MMI format as effective for evaluating applicants, and most (59% of candidates, 84% of interviewers, 77% of committee members) agreed that the MMI format should be continued. Cost per candidate interviewed was $136.34 (standard interview) vs $75.30 (MMI). Perceptions of the MMI process were favorable across stakeholder groups, and this format was less costly per candidate interviewed.

Correlation between MMI performance and OSCE performance – a pilot study

ObjectiveThe multiple mini-interview (MMI) has been shown to have a positive correlation with early medical school performance, clerkship evaluations, and national licensing examinations. There is limited data on its predictive validity at the postgraduate level.MethodsSix hundred and nineteen internship candidates were interviewed using the MMI format by the internal medicine residency program of The Reading Health System, between September 2011 and February 2014. Fifty-two interns were recruited. Each intern participated in an objective structured clinical examination (OSCE) 3–4 months after the start of the program. The OSCE score of each intern was used as the independent variable to test the relationship with both the MMI interpersonal score and the MMI overall score.ResultsThere was a moderate positive correlation between the average MMI interpersonal score and the communication score on the OSCE, r=0.384, n=52, p=0.005, and a negligible relationship between the average MMI overall score and the communication score on the OSCE, r=0.175, n=52, p=0.214.ConclusionThe MMI is a useful tool for residency programs to assess interpersonal and communication skills prior to matriculation into residency training. This study provides evidence for its validity in assessing these competencies.

Multiple mini-interviews: same concept, different approaches.

Increasing numbers of educational institutions in the medical field choose to replace their conventional admissions interviews with a multiple mini-interview (MMI) format because the latter has superior reliability values and reduces interviewer bias. As the MMI format can be adapted to the conditions of each institution, the question of under which circumstances an MMI is most expedient remains unresolved. This article systematically reviews the existing MMI literature to identify the aspects of MMI design that have impact on the reliability, validity and cost-efficiency of the format. Three electronic databases (OVID, PubMed, Web of Science) were searched for any publications in which MMIs and related approaches were discussed. Sixty-six publications were included in the analysis. Forty studies reported reliability values. Generally, raising the number of stations has more impact on reliability than raising the number of raters per station. Other factors with positive influence include the exclusion of stations that are too easy, and the use of normative anchored rating scales or skills-based rater training. Data on criterion-related validities and analyses of dimensionality were found in 31 studies. Irrespective of design differences, the relationship between MMI results and academic measures is small to zero. The McMaster University MMI predicts in-programme and licensing examination performance. Construct validity analyses are mostly exploratory and their results are inconclusive. Seven publications gave information on required resources or provided suggestions on how to save costs. The most relevant cost factors that are additional to those of conventional interviews are the costs of station development and actor payments. The MMI literature provides useful recommendations for reliable and cost-efficient MMI designs, but some important aspects have not yet been fully explored. More theory-driven research is needed concerning dimensionality and construct validity, the predictive validity of MMIs other than those of McMaster University, the comparison of station types, and a cost-efficient station development process.

Evaluation of an Interview Process for Admission Into a School of Pharmacy

To evaluate the doctor of pharmacy (PharmD) admissions interview process at North Dakota State University (NDSU). Faculty pairs interviewed candidates using a standardized grading rubric to evaluate qualitative parameters or attributes such as ethics, relevant life and work experience, emotional maturity, commitment to patient care, leadership, and understanding of the pharmacy profession. Total interview scores, individual attribute domain scores, and the consistency and reliability of the interviewers were assessed. The total mean interview score for the candidate pool was 17.4 of 25 points. Mean scores for individual domains ranged from 2.3 to 3.0 on a Likert-scale of 0-4. Nine of the 11 faculty pairs showed no mean differences from their interview partner in total interview scores given. Evaluations by 8 of the 11 faculty pairs produced high interrater reliability. The current interview process is generally consistent and reliable; however, future improvements such as additional interviewer training and adoption of a multiple mini-interview format could be made.

Enhancing the Reliability of the Multiple Mini-Interview for Selecting Prospective Health Care Leaders

The David Geffen School of Medicine at UCLA Program in Medical Education (UCLA-PRIME) used a 12-station multiple mini-interview (MMI) circuit to assess applicants. The authors sought to determine the reliability of the MMI, potential bias in scores, and the degree of acceptance by interviewers and applicants. In 2009, 28 interviewers interviewed a cohort of 76 applicants. An anonymous survey assessed interviewers' and applicants' satisfaction with the MMI process and perceived bias. Psychometric properties were determined with generalizability and decision theory. The process was repeated the following year with a new cohort of 78 applicants and minor modifications aimed at improving reliability. The MMI format was well received by both applicants and interviewers. No bias based on gender or disadvantaged status was found. The preliminary reliability of the MMI in 2009 was 0.58-lower than reported in previous studies-but improved in 2010 to 0.71 after an easy station was replaced with a more challenging one and a new scoring rubric was introduced. This interview technique proved to be reliable and was seen as transparent, uniform, and fair. The predictive validity of this process remains to be determined.

A Comparison of Behavioral and Multiple Mini-Interview Formats in Physician Assistant Program Admissions

The interview remains a widely used tool in health professions program admissions. The purpose of this study was to compare the use of a behavioral interview format with the multiple mini-interview format in measuring desired noncognitive behaviors. This dual cohort, observational, comparative study used a polytomous rating-scale model to analyze the results from two homogeneous groups of physician assistant (PA) program applicants (total N = 176). One group (n = 93) participated in two 20-minute behavioral interviews conducted by two raters per interviewee. The behavioral format included questions related to past behaviors and performance as a way to identify latent professionalism characteristics. The second group (n = 83) completed ten separate 7-minute stations with one rater per station. Each of the mini-stations assigned a specific topic and/or task to be completed. The score distributions related to applicant performance and station difficulty were plotted using Rasch analysis software. The behavioral interview format and multiple mini-interview had similar model fit. The behavioral interview did not adequately measure differences in applicant characteristics. In contrast, the multiple mini-interview measured more variation in noncognitive traits and identified better matching of station difficulty and person ability. In this study the multiple mini-interview format was a more reliable admissions tool in detecting latent professionalism attributes among PA program applicants. The multiple mini-interview format appeared to measure professional potential and organizational fit better than the behavioral interview format. A larger study across several programs may provide additional support for these findings.

Interviewer bias in medical student selection

To investigate whether interviewer personality, sex or being of the same sex as the interviewee, and training account for variance between interviewers' ratings in a medical student selection interview. In 2006 and 2007, data were collected from cohorts of each year's interviewers (by survey) and interviewees (by interview) participating in a multiple mini-interview (MMI) process to select students for an undergraduate medical degree in Australia. MMI scores were analysed and, to account for the nested nature of the data, multilevel modelling was used. Interviewer ratings; variance in interviewee scores. In 2006, 153 interviewers (94% response rate) and 268 interviewees (78%) participated in the study. In 2007, 139 interviewers (86%) and 238 interviewees (74%) participated. Interviewers with high levels of agreeableness gave higher interview ratings (correlation coefficient [r] = 0.26 in 2006; r = 0.24 in 2007) and, in 2007, those with high levels of neuroticism gave lower ratings (r = -0.25). In 2006 but not 2007, female interviewers gave higher overall ratings to male and female interviewees (t = 2.99, P = 0.003 in 2006; t = 2.16, P = 0.03 in 2007) but interviewer and interviewee being of the same sex did not affect ratings in either year. The amount of variance in interviewee scores attributable to differences between interviewers ranged from 3.1% to 24.8%, with the mean variance reducing after skills-based training (20.2% to 7.0%; t = 4.42, P = 0.004). This study indicates that rating leniency is associated with personality and sex of interviewers, but the effect is small. Random allocation of interviewers, similar proportions of male and female interviewers across applicant interview groups, use of the MMI format, and skills-based interviewer training are all likely to reduce the effect of variance between interviewers.

Rater and occasion impacts on the reliability of pre-admission assessments

Some medical schools have recently replaced the medical school pre-admission interview (MSPI) with the multiple mini-interview (MMI), which utilises objective structured clinical examination (OSCE)-style measurement techniques. Their motivation for doing so stems from the superior reliabilities obtained with the OSCE-style measures. Other institutions, however, are hesitant to embrace the MMI format because of the time and costs involved in restructuring recruitment and admission procedures. To shed light on the aetiology of the MMI's increased reliability and to explore the potential of an alternative, lower-cost interview format, this study examined the relative contributions of two facets (raters, occasions) to interview score reliability. Institutional review board approval was obtained to conduct a study of all students who completed one or more MSPIs at a large Midwestern medical college during 2003-2007. Within this dataset, we identified 168 applicants who were interviewed twice in consecutive years and thus provided the requisite data for generalisability (G) and decision (D) studies examining these issues. Increasing the number of interview occasions contributed much more to score reliability than did increasing the number of raters. Replicating a number of interviews, each with one rater, is likely to be superior to the often recommended panel interview approach and may offer a practical, low-cost method for enhancing MSPI reliability. Whether such a method will ultimately enhance MSPI validity warrants further investigation.

Assessment of Applicants to the Veterinary Curriculum Using a Multiple Mini-Interview Method

This study describes the development, implementation, and psychometric assessment of the multiple mini-interview (MMI) for the inaugural class of veterinary medicine applicants at the University of Calgary Faculty of Veterinary Medicine (UCVM). The MMI is a series of approximately five to 12 10-minute interviews that consist of situational events. Applicants are given a scenario and asked to work through an issue or behavioral-type questions that are meant to assess one attribute (e.g., empathy) at a time. This structure allows for multiple assessments of the applicants by trained interviewers on the same questions. MMI scenario development was based on a review of the noncognitive attributes currently assessed by the 31 veterinary schools across Canada and the United States and the goals and objectives of UCVM. The noncognitive attributes of applicants (N=110) were assessed at five stations, by two interviewers within each station, on three items using a standardized rating form on an anchored 1-5 scale. The method was determined to be reliable (G-coefficient=0.88) and demonstrated evidence of validity. The MMI score did not correlate with grade-point average (r=0.12, p=0.22). While neither the applicants nor interviewers had participated in an MMI format before, both groups reported the process to be acceptable in a post-interview questionnaire. This analysis provides preliminary evidence of the reliability, validity, and acceptability of the MMI in assessing the noncognitive attributes of applicants for veterinary medical school admissions.

Multiple mini-interviews: opinions of candidates and interviewers

To assess candidates' and interviewers' perceptions of the use of a multiple mini-interview (MMI) for selection of senior house officers (SHOs) to a UK regional paediatric training programme. Both candidates and interviewers completed anonymous questionnaires (comprising 16 and 25 questions, respectively). Demographic data were recorded for both groups. Data were analysed by frequencies; using Mann-Whitney and Kruskall-Wallis tests for comparisons; and Cronbach's alpha for internal consistency within the data. Both candidates and interviewers were positive about the fairness of the MMI (mean scores of 4.0 and 4.4, respectively). The majority of candidates (83%) had not been to this type of interview before. Gender, age and previous experience of MMIs did not account for differences in candidate responses (P > 0.05). A total of 86% of candidates were international medical graduates who preferred the format more than UK graduates did (P = 0.01). Interviewers were mainly experienced consultants who agreed that the multi-station format was better than the traditional interview (mean score 4.8) and represented a reliable process (mean score 4.4). Interviewers were concerned about the range of competencies covered and the subsequent performance of candidates in post (mean scores 3.6 and 3.2, respectively). Both candidates and interviewers agreed that the MMI format was reliable, fair and asked appropriate, easy-to-understand questions. In high-stakes interviews such as for specialty training in Modernising Medical Careers programmes, it is vital that all concerned have confidence in the selection process.