While adaptation to complex auditory stimuli has traditionally been reported for linguistic properties of speech, the present study demonstrates non-linguistic high-level aftereffects in the perception of voice identity, following adaptation to voices or faces of personally familiar speakers. In Exp. 1, prolonged exposure to speaker A′s voice biased the perception of identity-ambiguous voice morphs between speakers A and B towards speaker B (and vice versa). Significantly biased voice identity perception was also observed in Exp. 2 when adaptors were videos of speakers’ silently articulating faces, although effects were reduced in magnitude relative to those seen in Exp. 1. By contrast, adaptation to an unrelated speaker C elicited an intermediate proportion of speaker A identifications in both experiments. While crossmodal aftereffects on auditory identification ( Exp. 2) dissipated rapidly, unimodal aftereffects ( Exp. 1) were still measurable a few minutes after adaptation. These novel findings suggest contrastive coding of voice identity in long-term memory, with at least two perceptual mechanisms of voice identity adaptation: one related to auditory coding of voice characteristics, and another related to multimodal coding of familiar speaker identity.