Prior MRI studies have shown that patients with subcortical ischemic vascular disease (SIVD) exhibited white matter damage, gray matter atrophy and memory impairment, but the specific characteristics and interrelationships of these abnormal changes have not been fully elucidated. We collected the MRI data and memory scores from 29 SIVD patients with cognitive impairment (SIVD-CI), 29 SIVD patients with cognitive unimpaired (SIVD-CU) and 32 normal controls (NC). Subsequently, the thicknesses and volumes of the gray matter regions that are closely related to memory function were automatically assessed using FreeSurfer software. Then, the volume, fractional anisotropy (FA), mean diffusivity (MD), amplitude of low-frequency fluctuation (ALFF) and regional homogeneity (ReHo) values of white matter hyperintensity (WMH) region and normal-appearing white matter (NAWM) were obtained using SPM, DPARSF, and FSL software. Finally, the analysis of covariance, spearman correlation and mediation analysis were used to analyze data. Compared with NC group, patients in SIVD-CI and SIVD-CU groups showed significantly abnormal volume, FA, MD, ALFF, and ReHo values of WMH region and NAWM, as well as significantly decreased volume and thickness values of gray matter regions, mainly including thalamus, middle temporal gyrus and hippocampal subfields such as cornu ammonis (CA) 1. These abnormal changes were significantly correlated with decreased visual, auditory and working memory scores. Compared with the SIVD-CU group, the significant reductions of the left CA2/3, right amygdala, right parasubiculum and NAWM volumes and the significant increases of the MD values in the WMH region and NAWM were found in the SIVD-CI group. And the increased MD values were significantly related to working memory scores. Moreover, the decreased CA1 and thalamus volumes mediated the correlations between the abnormal microstructure indicators in WMH region and the decreased memory scores in the SIVD-CI group. Patients with SIVD had structural and functional damages in both WMH and NAWM, along with specific gray matter atrophy, which were closely related to memory impairment, especially CA1 atrophy and thalamic atrophy. More importantly, the volumes of some temporomesial regions and the MD values of WMH regions and NAWM may be potentially helpful neuroimaging indicators for distinguishing between SIVD-CI and SIVD-CU patients.