Evidence from everyday life suggests that differences in social behaviors between males and females exist, both in animal and in humans. These differences can be related to socio-cultural determinants, but also to specialized portions of the brain (the social brain), from the neurotransmitter to the neural network level. The high vulnerability of this system is expressed by the wide range of neuropsychiatric disorders associated with social dysfunctions, particularly social withdrawal. The principal psychiatric disorders with prominent social withdrawal are described, including hikikomori-like syndromes, and anxiety, depressive, autistic, schizophrenic, and personality disorders. It is hypothesized that social withdrawal can be partially independent from other symptoms and likely reflect alterations in the social brain itself, leading to a similar, transdiagnostic social dysfunction, reflecting defects in the social brain across a variety of psychopathological conditions. An overview is provided of gender effects in the biological determinants of social behavior, including: the anatomical structures of the social brain; the dimorphic brain structures, and the modulation of their development by sex steroids; gender differences in "social" neurotransmitters (vasopressin and oxytocin), and in their response to social stress. A better comprehension of gender differences in the phenotypes of social disorders and in the neural bases of social behaviors may provide new insights for timely, focused, innovative, and gender-specific treatments.