Method factor due to the use of reverse-keyed items: Is it simply a response style artifact?

Abstract

The method factor is usually conceptualized as the unexplained variance due to the use of more than one method to measure a construct. In the context of item keying direction, it often represents unique variance due to the use of reverse-keyed items. Alessandri et al. (Multivariate Behavioral Research, 46(4), 625–642, 2011) found that method factors from self- and peer ratings were moderately correlated with each other, but their investigation was limited to one measurement scale (optimism). In addition, their putative method factor was possibly measuring a substantive construct (pessimism) rather than simple method-related variance. The current study aimed to improve on their investigation by using multiple scales, each of which is theoretically unidimensional. Surprisingly, we replicated their finding that method factors were moderately correlated between the two types of raters. Contrary to Alessandri et al.’s finding, the self-rating method factor was not correlated with personality variables but was correlated with participants’ social desirability, whereas the peer-rating method factor was correlated with closeness to and duration of relationship with participants, but not with social desirability. Therefore, contrary to the explanation by Alessandri et al. (Multivariate Behavioral Research, 46(4), 625–642, 2011), it is unlikely that the method factor from peer ratings is due to social desirability. Our findings cast doubt on the view that cognitive abilities are the only explanation of the method factors. Contrary to the recent argument by Gnambs and Schroeders (Assessment 27(2), 404-441, 2020), method factors may not be a simple “response style artifact”.