Abstract
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. However, the reduced precision of trait estimates in this questionnaire format is an important drawback. Accordingly, this article presents an optimization procedure for assembling pairwise forced-choice questionnaires while maximizing posterior marginal reliabilities. This procedure is performed through the adaptation of a known genetic algorithm (GA) for combinatorial problems. In a simulation study, the efficiency of the proposed procedure was compared with a quasi-brute-force (BF) search. For this purpose, five-dimensional item pools were simulated to emulate the real problem of generating a forced-choice personality questionnaire under the five-factor model. Three factors were manipulated: (1) the length of the questionnaire, (2) the relative item pool size with respect to the questionnaire’s length, and (3) the true correlations between traits. The recovery of the person parameters for each assembled questionnaire was evaluated through the squared correlation between estimated and true parameters, the root mean square error between the estimated and true parameters, the average difference between the estimated and true inter-trait correlations, and the average standard error for each trait level. The proposed GA offered more accurate trait estimates than the BF search within a reasonable computation time in every simulation condition. Such improvements were especially important when measuring correlated traits and when the relative item pool sizes were higher. A user-friendly online implementation of the algorithm was made available to the users.
Highlights
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios
The average of the trait recovery criteria across the candidate questionnaires in each condition served as an indicator of the expected accuracy of a randomly assembled questionnaire for a given item pool. This indicator aims to represent the results for forced-choice questionnaires (FCQs) built using some structural criteria, which is a common practice in current research involving the forced-choice format
The initial solutions were similar with both procedures, the genetic algorithm (GA) rapidly overtook the best candidates formed with the constrained random assembly in the BF search
Summary
The use of multidimensional forced-choice questionnaires has been proposed as a means of improving validity in the assessment of non-cognitive attributes in high-stakes scenarios. The use of multidimensional forced-choice questionnaires (FCQs) has been proposed to prevent these response biases in the assessment of non-cognitive domains (e.g., Cao & Drasgow, 2019; Cheung & Chan, 2002; Salgado & Táuriz, 2014; Wetzel et al, 2021), as they offer comparable or better convergent and criterion-related validity (e.g., Kreitchmann et al, 2019; Otero et al, 2020) This format differs from rating scales, in that instead of indicating one’s agreement with a statement on an ordinal scale, respondents must rank two or more statements within a block according to their agreement with each statement. If statements have similar social desirability, SDR and faking will be harder to engage with (Lee & Joo, 2021; Wetzel et al, 2021)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
