Loneliness has an important impact on memory function in late life. However, the neural mechanism by which loneliness detrimentally influences memory function remains elusive. Furthermore, it remains unclear whether the association between loneliness and memory function varies by gender. The current study aimed to investigate the neural mechanism underlying the association between loneliness and episodic memory function and explore whether it varies with gender among cognitively normal older adults. A total of 173 community-dwelling adults aged 60 years or older from the Korean Social Life, Health, and Aging Project (KSHAP) study (mean age = 71.87) underwent an assessment of loneliness, neuropsychological testing, and structural magnetic resonance imaging. The association between loneliness and episodic memory function was mediated by the volume of white matter hyperintensities (WMHs), but not by hippocampal or gray matter volumes. In addition, the association between loneliness and memory function through WMHs was significantly moderated by gender; specifically, the indirect effect was significant among men but not among women. The study suggests that WMHs may be a potential neurological mechanism that causes late-life memory dysfunction associated with loneliness in older men. The findings underscore the need for gender-specific interventions to mitigate memory impairment associated with late-life loneliness, with significant public health implications.