Two experiments with monolingual and bilingual participants tested memory for sources (speakers) and destinations (listeners) in conversations consisting of self-referential statements. In Experiment 1, participants directly interacted in English conversations with audio-visually recorded confederates. In Experiment 2, participants observed recorded conversations among confederates. In both conversational situations, source memory was more accurate than destination memory, indicating that the attentional resources consumed by self-focus or sentence production/completion do not explain why destinations are less well remembered than sources in direct-interaction conversations. Source and destination memory were positively associated with item memory at the participant level, indicating that stronger item encoding is associated with stronger encoding of contextual information. In the observed conversations, source and destination accuracy were negatively associated at the trial level, indicating that these features of the memory episode are not encoded independently, and there is a tradeoff in the encoding of these contextual features. Item memory did not differ for monolinguals and bilinguals and was positively associated with proficiency only in conversations with direct interaction. In the observational setting (but not the direct-interaction setting), source and destination memory were more accurate for bilinguals than monolinguals. This finding suggests that bilinguals allocate attention more efficiently than monolinguals when the cognitive demands of sentence production are eliminated. Proficiency in English was positively associated with memory for the appropriate conversational partner only when participants had to produce sentence frames and complete them with self-generated information, suggesting that language proficiency is beneficial when cognitive demands are high.