Social Facilitation/Inhibition (SFI) refers to how others' presence influences task performance positively or negatively. Our previous study revealed that peer presence modulated saccadic eye movements, a fundamental sensorimotor activity. Pro- and anti-saccades were either facilitated or inhibited depending on trial block complexity (Tricoche et al., 2020). In the present fMRI study, we adapted our paradigm to investigate the neural basis of SFI on saccades. Considering inter- and intra-individual variabilities, we evaluated the shared and distinct neural patterns between social facilitation and inhibition. We predicted an involvement of the saccade-related and attention networks, alongside the Theory-of-Mind (ToM) network, with opposite activity changes between facilitation and inhibition. Results confirmed peer presence modulation in fronto-parietal areas related to saccades and attention, in opposite directions for facilitation and inhibition. Additionally, the ventral attention network was modulated during inhibition. Default mode regions, including ToM areas, were also modulated. Finally, pupil size, often linked to arousal, increased with peers and correlated with dorsal attention regions and anterior insula activities. These results suggest that SFI engages task-specific and domain-general networks, modulated differently based on observed social effect. Attention network seemed to play a central role at both basic (linked to arousal or vigilance) and cognitive control levels.