ROCs in rats? Response to Wixted and Squire: Figure 1.

Abstract

The article, “Constructing receiver operating characteristics (ROCs) with experimental animals: Cautionary Notes” offered by Wixted and Squire (2008, this issue) attempts to dismiss our observation of linear ROC functions in rats performing recognition tasks in two recent reports (Fortin et al. 2004; Sauvage et al. 2008). The motive for their effort is that the model of recognition memory they choose to adopt (the single-process model of recognition memory known as the Unequal Variance Signal Detection Model) cannot account for a linear ROC, so they are convinced that there must be a violation of key assumptions in signal detection analysis methodology that artificially causes the functions to be linear. Of note, our approach has been different; we examined our data using both the singleand dual-process models and found that the dual-process model provided a better fit in both studies. Wixted and Squire begin with the assertion that linear ROC functions are almost never observed in humans. However, their analysis focuses on the linearity of the probability ROCs, which rests only on the failure to find statistically significant curvilinearity. More compelling are analyses that calculate performance in z-scores, which reveal a statistically significant U-shape curve in z-space that provides direct evidence for ROC linearity. On those grounds, our ROCs are in perfect agreement with the vast majority of the findings from associative recognition tests (59 conditions from 17 different studies) (Parks and Yonelinas 2007), wherein a large majority showed the U-shaped zROCs as seen in our studies. U-shaped zROC functions were observed across a variety of different associative recognition tasks, including tests of word pairs, memory for location, list membership, and more. Also, Wixted and Squire emphasize that our use of differential reward payoffs to manipulate response biases is unique and suspect. In contrast to this claim, the manipulation of payoff ratios is considered a valid way to obtain a range of response criteria to determine ROCs, and this method has been used since the beginning of signal detection analyses. The use of payoff ratios was initially used in signal detection analyses on visual detection performance (Tanner and Swets 1954) and produced curvilinear ROCs. In their comprehensive review of the literature, Macmillan and Creelman (1991) conclude: “Payoffs and verbal instructions, although more time consuming than ratings provide one important advantage: The data points are statistically independent, as they are not in a rating-experiment ROC. If the aim of an experiment is to evaluate theoretical assumptions (e.g., that the ROC is regular), then a separate-session procedure may be indicated. For most practical applications, however, the rating task is recommended because of its efficiency.” Therefore, while confidence ratings are more convenient for studies on humans, the use of payoff ratios is considered valid and does not necessarily produce linear ROCs in humans. Indeed, in our particular application of payoff ratios, we observed curvilinear ROCs under some memory demands and linear ROCs under others, using identical payoff ratio conditions for all conditions (see below). Therefore, while we appreciate Wixted and Squire’s suggestion of a different method that has been used to manipulate biases, the literature provides many precedents for our use of payoff ratios as equally valid. We will not further argue these general issues here, except to highlight that Wixted and Squire’s interpretation of the literature on the shape of ROC functions and on the use of payoff ratios is far from consensual. Rather, we will focus on their three main criticisms: that a valid ROC requires equal accuracy across bias levels, a “differential outcomes effect” explains the observations of linear ROCs, and other aspects of our protocol might force the ROC curve to be linear.