The internal face prototype is thought to be a construction of the average of every previously viewed face (Schwaninger et al., 2003). However, the influence of the most frequently encountered faces (i.e., personally familiar faces) has been generally understated. The current research explored the face distortion aftereffect in unfamiliar, famous, and personally familiar (each subject’s parent) faces. Forty-eight adult participants reported whether faces were distorted or not (distorted by shifting the eyes in the vertical axis) of a series of images that included unfamiliar, famous, and personally familiar faces. The number of faces perceived to be “odd” was measured pre- and post-adaptation to the most extreme distortion. Participants were adapted to either an unfamiliar, famous, or personally familiar face. The results indicate that adaptation transferred from unfamiliar faces to personally familiar faces more so than the converse and aftereffects did not transfer from famous faces to unfamiliar faces. These results are indicative of representation differences between unfamiliar, famous, and personally familiar faces, whereby personally familiar faces share representations of both unfamiliar and famous faces.