Purpose To investigate associations between neuroimaging markers of cerebrovascular disease, including lesion topography and extent and severity of strategic and global cerebral tissue injury, and cognition in carotid artery disease (CAD). Materials and Methods All participants gave written informed consent to undergo brain magnetic resonance imaging and the Addenbrooke's Cognitive Examination-Revised. One hundred eight patients with symptomatic CAD but no dementia were included, and a score less than 82 represented cognitive impairment. Group comparison and interrelations between global cognitive and fluency performance, lesion topography, and ultrastructural damage were assessed with voxel-based statistics. Associations between cognition, medial temporal lobe atrophy (MTA), lesion volumes, and global white matter ultrastructural damage indexed as increased mean diffusivity were tested with regression analysis by controlling for age. Diagnostic accuracy of imaging markers selected from a multivariate prediction model was tested with receiver operating characteristic analysis. Results Cognitively impaired patients (n = 53 [49.1%], classified as having probable vascular cognitive disorder) were older than nonimpaired patients (P = .027) and had more frequent MTA (P < .001), more cortical infarctions (P = .016), and larger volumes of acute (P = .028) and chronic (P = .009) subcortical ischemic lesions. Lesion volumes did not correlate with global cognitive performance (lacunar infarctions, P = .060; acute lesions, P = .088; chronic subcortical ischemic lesions, P = .085). In contrast, cognitive performance correlated with presence of chronic ischemic lesions within the interhemispheric tracts and thalamic radiation (P < .05, false discovery rate corrected). Skeleton mean diffusivity showed the closest correlation with cognition (R2 = 0.311, P < .001) and promising diagnostic accuracy for vascular cognitive disorder (area under the curve, 0.82 [95% confidence interval: 0.75, 0.90]). Findings were confirmed in subjects with a low risk of preclinical Alzheimer disease indexed by the absence of MTA (n = 85). Conclusion Subcortical white matter ischemic lesion locations and severity of ultrastructural tract damage contribute to cognitive impairment in symptomatic CAD, which suggests that subcortical disconnection within large-scale cognitive neural networks is a key mechanism of vascular cognitive disorder. Online supplemental material is available for this article.