Abstract
Factorial survey experiments have been widely used to study recruiters’ hiring intentions. Respondents are asked to evaluate hypothetical applicant descriptions, which are experimentally manipulated, for hypothetical job descriptions. However, this methodology has been criticized for putting respondents in hypothetical situations that often only partially correspond to real-life hiring situations. It has been proposed that this criticism can be overcome by sampling real-world vacancies and the recruiters responsible for filling them. In such an approach, only the applicants’ descriptions are hypothetical; respondents are asked about a real hiring problem, which might increase internal and external validity. In this study, we test whether using real vacancies triggers more valid judgments compared to designs based on hypothetical vacancies. The growing number of factorial survey experiments conducted in employer studies makes addressing this question relevant, both for methodological and practical reasons. However, despite the potential implications for the validity of data, it has been neglected so far. We conducted a factorial survey experiment in Luxembourg, in which respondents evaluated hypothetical applicants referring either to a currently vacant position in their company or to a hypothetical job. Overall, we found little evidence for differences in responses by the design of the survey experiment. However, the use of real vacancies might prove beneficial depending on the research interest. We hope that our comparison of designs using real and hypothetical vacancies contributes to the emerging methodological inquiry on the possibilities and limits of using factorial survey experiments in research on hiring.
Highlights
Factorial survey experiments (FSEs) have been used in the social sciences to study human judgments since the 1970s (Rossi 1979; Wallander 2009) and, in recent years, have become an established method to study how employers select job applicants (McDonald 2019)
Regarding information technology (IT) jobs (Fig. 2b), we found no effect of migrant background in either FSE version, contradicting the benchmark; while the effects tended to be negative in the real vacancy” (RV) version and positive in the hypothetical vacancy” (HV) version, the difference was not statistically significant
We conducted an FSE for two occupational fields in Luxembourg to examine whether the effects of applicant characteristics on observed hiring intentions differ when the vignette ratings refer to a real instead of a hypothetical vacancy
Summary
Factorial survey experiments (FSEs) have been used in the social sciences to study human judgments since the 1970s (Rossi 1979; Wallander 2009) and, in recent years, have become an established method to study how employers select job applicants (McDonald 2019). FSEs have been criticized for measuring the hiring intentions of employers, rather than their real behavior (e.g., Pager and Quillian 2005). Field experiments, such as correspondence or audit studies in which fictitious applications are sent to real jobs (e.g., Pedulla 2018; Protsch and Solga 2014), are often seen as better suited to detecting inequalities in the hiring process.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
