A Many-faceted Rasch Model Analysis of Structured Interview

Abstract

Being one of the most important techniques in personnel selection, structured interview has attracted more and more research interest in improving its reliability and validity. Some researches focus on the standardization of its content and dimensions, others try to decrease rater bias by intensive rater training. The third one, to handle the possible bias in statistic way, has attracted more and more attention. Many-faceted Rasch Model (MFRM), an extension to Rasch model, served as such kind of techniques. By parameterizing not only interviewee's ability and item difficulty but also judge severity, MFRM offers an effective way to estimate interviewee's latent trait, that is, the ability, and provides detailed information of inter-rater reliability as far as a specific interviewee is concerned. This study used MFRM to analyze the result of a structured interview and demonstrated a creative way to locate the source of bias for a specific interviewee. Data came from a structured interview. There were 7 raters in each interview panel. Since the interview last two days, these 21 raters were randomized into 3 panels in the morning of each day in order to prevent cheating. Rating scores of two panels were used in this study. A、B、C、D、E、F、G were raters of one panel and interviewed interviewees numbered 1~34. A、E、H、I、J、K、L were raters in the other panel that interviewees numbered 35~66 were interviewed. Each rater rated each interviewee independently on five dimensions using a 10 points rating scale. Using Facets 3.62.0, a computer program based on MFRM, the abilities of 66 interviewees was estimated, accompanied with a Infit MnSq, which demonstrated the degree to which raters in the panel agreed with each other on the evaluation of a specific interviewee. The ranking order based on interview raw scores and Facets estimated logits values were compared. Difference were found between those them. To track the source of error for interviewee numbered 56, bias analysis of Facets was also made. The ability of 66 interviewees were reported with infit MnSq, showing inter-rater reliability at individual level; The ranking order based on interview raw score and estimated ability score were quite different, especially for some interviews. Taking interviewee numbered 56 for example, the ranking difference was as large as 15. Bias analysis aimed at locating the source of error for interviewee number 56 show that not only rater consistency, but also rater severity contribute to the ranking difference. The results confirmed the utility of MFRM analysis. The application of MFRM in the analysis of structured interview was proved to be not only an effective way in personnel selection, but also provided diagnostic information for sources of error locating at individual level.