Cerebral Small Vessel Disease (SVD); lacunes, Cerebral Microbleeds (CMBs), and White Matter Hyperintensities (WMH) have a vital role in cognitive impairment and dementia. SVD in lobar location is related to cerebral amyloid angiopathy, whereas SVD in a deep location with hypertensive arteriopathy. It remains unclear how different locations of SVD affect long-term cognitive decline. The present study aimed to analyse the association between different locations and severity of SVD with global and domain-specific cognitive decline over the follow-up interval of 3 years. We studied 428 participants who had performed MRI scans at baseline and at least 3 neuropsychological assessments. Locations of lacunes and CMBs were categorized into strictly lobar, strictly deep and mixed-location, WMH volume into anterior and posterior. The National Institute of Neurological Disorders and Stroke-Canadian Stroke Network Harmonization Neuropsychological Battery was used to assess cognitive function. To analyse the association between baseline location and severity of SVD with cognitive decline, linear regression models with generalized estimated equations were constructed to calculate the mean difference, 95% confidence interval and two-way interaction factor between time and SVD. Increased numbers of baseline CMBs were associated with a decline in global cognition as well as a decline in executive function and memory domains. Location-specific analysis showed similar results with strictly lobar CMBs. There was no association with strictly deep and mixed-location CMBs with cognitive decline. Baseline WMH volume was associated with a decline in global cognition, executive function and memory. Similar results were obtained with anterior and posterior WMH volumes. Lacunes and their locations were not associated with cognitive decline. Strictly lobar CMBs, as well as WMH volume in anterior and posterior regions, were associated with cognitive decline. Future research focuses are warranted to evaluate interventions that may prevent cognitive decline related to SVD.