Previous research has shown that retrospective gaze cues direct attention to internally maintained representations in visual working memory (vWM). Here, we aimed to differentiate the dual nature of gaze and accordingly proposed two hypotheses regarding the gaze-induced prioritization in vWM. The directional cueing hypothesis claims a constant attentional shifting to the gazed-at direction. By contrast, the referential cueing hypothesis proposes that gaze cues selectively orient attention toward their referents. To test these hypotheses, we employed an adapted change-detection task wherein gaze cues were presented during the retention interval. Critically, the cue character was positioned between two barriers, which could be either opaque (the blocked condition) or transparent (the unblocked condition). Polygons previously presented at the gazed-at (vs. gazed-away) location were better memorized, but not when the visual perspective of the character was obstructed (i.e., the blocked condition, Experiment 1). Subsequent experiments demonstrated that physical motion cues (Experiment 2) and inverted face cues (Experiment 3), which disrupted the extraction of referential signals, were immune to barrier settings. In Experiment 4, we generalize this selective cueing effect to faces with fearful expressions. These consistent findings support the referential cueing hypothesis and emphasize the distinctiveness of social attention. (PsycInfo Database Record (c) 2023 APA, all rights reserved).