Unsuccessful retrieval of proper names (PNs) is commonly observed in patients suffering from neurological conditions such as stroke or epilepsy. While a large body of works has suggested that PN retrieval relies on a cortical network centered on the left anterior temporal lobe (ATL), much less is known about the white matter connections underpinning this process. Sparse studies provided evidence for a possible role of the uncinate fasciculus, but the inferior longitudinal fasciculus (ILF) might also contribute, since it mainly projects into the ATL, interconnects it with the posterior lexical interface and is engaged in common name (CN) retrieval. To ascertain this hypothesis, we assessed 58 patients having undergone a neurosurgery for a left low-grade glioma by means of a famous face naming (FFN) task. The behavioural data were processed following a multilevel lesion approach, including location-based analyses, voxel-based lesion-symptom mapping (VLSM) and disconnection-symptom mapping. Different statistical models were generated to control for sociodemographic data, familiarity, biographical knowledge and control cognitive performances (i.e., semantic and episodic memory and CN retrieval). Overall, VLSM analyses indicated that damage to the mid-to-anterior part of the ventro-basal temporal cortex was especially associated with PN retrieval deficits. As expected, tract-oriented analyses showed that the left ILF was the most strongly associated pathway. Our results provide evidence for the pivotal role of the ILF in the PN retrieval network. This novel finding paves the way for a better understanding of the pathophysiological bases underlying PN retrieval difficulties in the various neurological conditions marked by white matter abnormalities.