Neuropsychological and neuroimaging studies have demonstrated a role for the amygdala in processing the perceived trustworthiness of faces, but it remains uncertain whether its responses are linear (with the greatest response to the least trustworthy-looking faces), or quadratic (with increased fMRI signal for the dimension extremes). It is also unclear whether the trustworthiness of the stimuli is crucial or if the same response pattern can be found for faces varying along other dimensions. In addition, the responses to perceived trustworthiness of face-selective regions other than the amygdala are seldom reported. The present study addressed these issues using a novel set of stimuli created through computer image-manipulation both to maximise the presence of naturally occurring cues that underpin trustworthiness judgments and to allow systematic manipulation of these cues. With a block-design fMRI paradigm, we investigated neural responses to computer-manipulated trustworthiness in the amygdala and core face-selective regions in the occipital and temporal lobes. We asked whether the activation pattern is specific for differences in trustworthiness or whether it would also track variation along an orthogonal male–female gender dimension. The main findings were quadratic responses to changes in both trustworthiness and gender in all regions. These results are consistent with the idea that face-responsive brain regions are sensitive to face distinctiveness as well as the social meaning of the face features.