The ventral temporal cortex (VTC) of the human cerebrum is critically engaged in computations related to high-level vision. One intriguing aspect of this region is its asymmetric organization and functional lateralization. Notably, in the VTC, neural responses to words are stronger in the left hemisphere, whereas neural responses to faces are stronger in the right hemisphere. Converging evidence has suggested that left-lateralized word responses emerge to couple efficiently with left-lateralized frontotemporal language regions, but evidence is more mixed regarding the sources of the right-lateralization for face perception. Here, we use individual differences as a tool to adjudicate between three theories of VTC organization arising from: 1) local competition between words and faces, 2) local competition between faces and other categories, 3) long-range coupling with VTC and frontotemporal areas subject to their own local competition. First, in an in-house functional MRI experiment, we demonstrated that individual differences in laterality are both substantial and reliable within a right-handed population of young adults. We found no (anti-)correlation in the laterality of word and face selectivity relative to object responses, and a positive correlation when using selectivity relative to a fixation baseline, challenging ideas of local competition between words and faces. We next examined broader local competition with faces using the large-scale Human Connectome Project (HCP) dataset. Face and tool laterality were significantly anti-correlated, while face and body laterality were positively correlated, consistent with the idea that generic local representational competition and cooperation may shape face lateralization. Last, we assessed the role of long-range coupling in the development of VTC laterality. Within our in-house experiment, substantial correlation was evident between VTC text laterality and several other nodes of a distributed text-processing circuit. In the HCP data, VTC face laterality was both negatively correlated with frontotemporal language laterality, and positively correlated with social perception laterality in the same areas, consistent with a long-range coupling effect between face and social processing representations, driven by local competition between language and social processing. We conclude that both local and long-range interactions shape the heterogeneous hemispheric specializations in high-level visual cortex.