Abstract The global population is aging rapidly, and a research question of critical importance is why some older adults suffer tremendous cognitive decline while others are mostly spared. Past aging research has shown that older adults with spared cognitive ability have better local short-range information processing while global long-range processing is less efficient. We took this research a step further to investigate whether the underlying structural connections, measured in vivo using diffusion magnetic resonance imaging (dMRI), show a similar shift to support cognitive ability. We analyzed the structural connectivity streamline probability (representing the probability of connection between regions) and nodal efficiency and local efficiency regional graph theory metrics to determine whether age and cognitive ability are related to structural network differences. We found that the relationship between structural connectivity and cognitive ability with age was nuanced, with some differences with age that were associated with poorer cognitive outcomes, but other reorganizations that were associated with spared cognitive ability. These positive changes included strengthened local intrahemispheric connectivity and increased nodal efficiency of the ventral occipital-temporal stream, nucleus accumbens, and hippocampus for older adults, and widespread local efficiency primarily for middle-aged individuals.