A central claim in research on interactive conversation is that listeners use the knowledge assumed to be shared with a conversational partner to guide their understanding of utterances from the earliest moments of processing. In the present study we investigated whether this claim extends to cases where shared vs. private knowledge is discrepant in terms of the identity assigned to a mutually seen object that could be misidentified on the basis of its appearance. Eye movement measures were used to evaluate listeners’ ability to integrate a speaker’s perspective as they identified the referent for an unfolding expression. The results reconfirmed previous findings showing that listeners can rapidly take into account a speaker’s awareness of the existence/presence of a referential object. In contrast, however, listeners showed strong consideration of their private knowledge about the identity of an object during referential processing. Strikingly, this tendency was found even when speaker-produced discourse reinforced the way in which the speaker’s understanding of the object’s identity differed from that of the listener. Together, the results reveal clear and important differences in the way in which distinct types of perspective-based cues are integrated in real-time communicative interaction.