Social categorization is known to be an important part of social cognition. The categorizations we use, despite their multitude, frequently take the form of the general ingroup/outgroup distinction. A meta-analysis of 33 fMRI studies, reporting selective activations to various social groups, was used to identify common neural structures responsible for relational representation of social structure. Activation Likelihood Estimation (ALE) analysis revealed areas in bilateral amygdala, cingulate gyrus, fusiform gyrus, right TPJ and right insula as implementing various aspects of social categorization. Activation of amygdala can be associated with modulation of behavioral response to subjectively significant stimuli. A more ventral part of anterior cingulate cortex (ACC) can be associated with self-referential reasoning about ingroup members while a more dorsal part of ACC is involved in the regulation of emotions toward outgroup members. Right insula can be engaged in the modulation of outgroup avoidance behavior. Fusiform gyrus (FG) appears to be directly involved in social categorization process via top-down modulation of social perception. Yet it is difficult to associate any of the revealed clusters with the relational ingroup/outgroup structure.