Comparison of Academic, Administrative and Community Rater Scores at a Multiple Mini Interview Using Generalisability Theory

Abstract

Multiple Mini Interviews (MMIs) are sampling approaches that use multiple short stations to select prospective students for professional programmes. Each station uses different interview scenarios and raters to effectively assess candidates’ noncognitive skills. This study compared the performances of three sets of raters; academic, administrative staff, and community members, in an MMI for student selection using performance comparisons and Generalisability Theory to estimate the different sources of variance and generalisability (reliability) coefficients. The study aims to analyse the differences in performance scores from these raters and their psychometric projections on reliability with different samples of raters and stations. Eleven candidates participated in the 10-station MMI, each with an eight-minute duration, two minutes of preparation, and an academic assessment using a marking rubric. The entire interview was video recorded. The administrative staff and community members watched the videos independently and graded all candidates’ performances using the same marking rubric. Generalisability and Decision studies were used to analyse the collected data. Community members were the strictest, while academics were the most lenient. There were statistically significant differences between rater categories in six stations. The generalisability coefficient of 0.85 of onerater results from the Decision study suggested good reliability of the 10-station MMI. The Decision study found that generalisability coefficients improved more with an increasing number of raters rather than number of stations. Four stations contributed to unreliability in each rater category and a combination of the rater categories. Information on number of stations, number of raters, and type of rater combination required to achieve good reliability enabled informed decisions on the process and implementation of the MMI. The station simulation that influenced unreliability helped us improve station writing and identify focus areas for training and development.