Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure

Abstract

Objective. To determine whether there are differences in the performance and quality of multiple-choice items with opposite stem orientations (positive or negative), based on a novel item quality measure and conventional psychometric parameters.Methods. A retrospective study was conducted on multiple-choice assessment items used in years two and three of pharmacy school for pharmacotherapy and related courses administered between August 2018 and December 2019. Conventional psychometric parameters (difficulty and discrimination indices), average response time, nonfunctional distractor percentage, and a novel measure of item quality of negatively worded items were compared with those of control items, namely positively worded items (n=103 each). This novel measure uses difficulty and discrimination in tandem for the decision to reject, review, or retain items in an assessment. Statistical analyses were performed on continuous and categorical variables, on the relationship between difficulty and discrimination, and on differences in correlation coefficients between positively and negatively worded items.Results. Stem orientation was not significantly associated with the novel measure of item quality. Also, there were no significant differences between positively and negatively worded items in any of the psychometric parameters. There were significant, negative correlations between difficulty and discrimination indices in both groups, and the correlation coefficients were significantly stronger in positively versus negatively worded items.Conclusion. Items with opposite stem orientations show no differences in the novel item quality measure nor in conventional measures of performance and quality, except in difficulty-discrimination relationships. This suggests that negatively worded items should be used when necessary, but cautiously.