Face imagery can access facial memories without the use of perceptual stimuli. Current data on the relation of imagery to the perceptual function and neuroanatomy of prosopagnosic patients are mixed, and little is known about the type of facial information patients can access through imagery. The authors wished to determine 1) which lesions abolished face imagery in prosopagnosia, 2) if deficits in perceiving facial structure were paralleled by similar deficits in imagery, and 3) if covert recognition of faces correlated with the degree of residual imagery for faces. The authors tested nine prosopagnosic patients who had been tested previously for perception of facial configuration and covert recognition of famous faces. The authors constructed a battery of 37 questions that asked subjects to imagine the faces of two celebrities and to choose which one had a certain facial property. Half were questions about facial features and half were about overall facial shape. Imagery was abolished only by anterior temporal lesions. Imagery for facial shape but not features was degraded by lesions of the right hemisphere's fusiform face area, which severely impaired perception of facial configuration. Feature imagery was degraded only when there was associated left occipito-temporal damage. Covert recognition was found when either configural perception or imagery was severely damaged, but not when both were abnormal. In patients with impaired configural perception, covert recognition correlated with feature imagery, suggesting that feature-based processing may drive residual covert abilities in these patients. Although anterior temporal cortex may be the site of facial memory stores, these data also support hypotheses that perceptual areas like the fusiform face area have parallel contributions to mental imagery. The data on covert recognition are consistent with a view that it is the residue of a partially damaged face-recognition network. Covert recognition may reflect the degree of damage across components of a network rather than mark a specific form of prosopagnosia or a dissociated pathway.