Silent or asymptomatic lacunar infarcts (LACI) are common in elderly individuals, but it remains largely unclear how these often neglected silent brain infarcts lead to multiple domain cognitive deficits and even Alzheimer's disease (AD). In this study, we investigated the difference between patients with silent LACI in basal ganglia region and healthy controls for the structural and functional changes in the aspects of alterations of gray matter (GM) volume and intra-/inter-default mode network (DMN) and salience network (SN) connectivity. Thirty patients with silent LACI in the basal ganglia region and thirty healthy controls participated in the study. Voxel-based morphometry analysis was employed to measure the GM volume. We further investigated the intra/inter-network connectivity of DMN and SN using resting-state functional magnetic resonance imaging. Compared with healthy controls, patients performed worse in cognitive function in the aspects of general mental status, attention, and memory. The LACIs showed more severe GM atrophy in insula, anterior cingulate cortex, caudate, and superior temporal pole than controls. The connectivity within and between two networks was also reduced in patients. Importantly, the disrupted connectivity correlated with the patients' cognitive performance. Our findings support the hypothesis that silent lacunar infarcts result in cognitive decline, GM, and functional connectivity loss.