Binocular disparity is represented by interocular cross-correlation of visual images in the striate and some extrastriate cortices. This correlation-based representation produces reversed depth perception in a binocularly anticorrelated random-dot stereogram (aRDS) when it is accompanied by an adjacent correlated RDS (cRDS). Removal of the cRDS or spatial separation between the aRDS and cRDS abolishes reversed depth perception. However, how an immediate plane supports reversed depth perception is unclear. One possible explanation is that the correlation-based representation generates reversed depth based on the relative disparity between the aRDS and cRDS rather than the absolute disparity of the aRDS. Here, we psychophysically tested this hypothesis. We found that participants perceived reversed depth in an aRDS with zero absolute disparity when it was surrounded by a cRDS with nonzero absolute disparity (i.e., nonzero relative disparity), suggesting a role of relative disparity on the depth reversal. In addition, manipulation of the absolute disparities of the central aRDS and surrounding cRDS caused depth perception to reverse with respect to the depth of the surround. Further, depth reversal persisted after swapping the locations of the two RDSs. A model of relative-disparity encoding explains all these results. We conclude that reversed depth perception in aRDSs occurs in a relative frame of reference and suggest that the visual system contains correlation-based representation that encodes relative disparity.