People with schizophrenia (SCZ) and bipolar disorder (BD) have impairments in processing social information, including faces. The neural correlates of face processing are widely studied with the N170 ERP component. However, it is unclear whether N170 deficits reflect neural abnormalities associated with these clinical conditions or differences in social environments. The goal of this study was to determine whether N170 deficits would still be present in SCZ and BD when compared with socially isolated community members. Participants included 66 people with SCZ, 37 with BD, and 125 community members (76 "Community-Isolated"; 49 "Community-Connected"). Electroencephalography was recorded during a face processing task in which participants identified the gender of a face, the emotion of a face (angry, happy, neutral), or the number of stories in a building. We examined group differences in the N170 face effect (greater amplitudes for faces vs buildings) and the N170 emotion effect (greater amplitudes for emotional vs neutral expressions). Groups significantly differed in levels of social isolation (Community-Isolated > SCZ > BD = Community-Connected). SCZ participants had significantly reduced N170 amplitudes to faces compared with both community groups, which did not differ from each other. The BD group was intermediate and did not differ from any group. There were no significant group differences in the processing of specific emotional facial expressions. The N170 is abnormal in SCZ even when compared to socially isolated community members. Hence, the N170 seems to reflect a social processing impairment in SCZ that is separate from level of social isolation.