Brain microstructural alterations possibly occur in the normal-appearing white matter (NAWM) and grey matter of small vessel disease (SVD) patients, and may contribute to cognitive impairment. The aim of this study was to explore cognitive related microstructural alterations in white matter and deep grey matter nuclei in SVD patients using magnetic resonance (MR) quantitative susceptibility mapping (QSM). 170 SVD patients, including 103 vascular mild cognitive impairment (VaMCI) and 67 no cognitive impairment (NCI), and 21 healthy control (HC) subjects were included, all underwent a whole-brain QSM scanning. Using a white matter and a deep grey matter atlas, subregion-based QSM analysis was conducted to identify and characterize microstructural alterations occurring within white matter and subcortical nuclei. Significantly different susceptibility values were revealed in NAWM and in several specific white matter tracts including anterior limb of internal capsule, corticospinal tract, medial lemniscus, middle frontal blade, superior corona radiata and tapetum among VaMCI, NCI and HC groups. However, no difference was found in white matter hyperintensities between VaMCI and NCI. A trend toward higher susceptibility in the caudate nucleus and globus pallidus of VaMCI patients compared to HC, indicating elevated iron deposition in these areas. Interestingly, some of these QSM parameters were closely correlated with both global and specific cognitive function scores, controlling age, gender and education level. Our study suggested that QSM may serve as a useful imaging tool for monitoring cognitive related microstructural alterations in brain. This is especially meaningful for white matter which previously lacks of attention.