Alzheimer's disease currently has no cure and is usually detected too late for interventions to be effective. In this study we have focused on cognitively normal subjects to study the impact of risk factors on their long-range brain connections. To detect vulnerable connections, we devised a multiscale, hierarchical method for spatial clustering of the whole brain tractogram and examined the impact of age and APOE allelic variation on cognitive abilities and bundle properties including texture e.g., mean fractional anisotropy, variability, and geometric properties including streamline length, volume, and shape, as well as asymmetry. We found that the third level subdivision in the bundle hierarchy provided the most sensitive ability to detect age and genotype differences associated with risk factors. Our results indicate that frontal bundles were a major age predictor, while the occipital cortex and cerebellar connections were important risk predictors that were heavily genotype dependent, and showed accelerated decline in fractional anisotropy, shape similarity, and increased asymmetry. Cognitive metrics related to olfactory memory were mapped to bundles, providing possible early markers of neurodegeneration. In addition, physiological metrics such as diastolic blood pressure were associated with changes in white matter tracts. Our novel method for a data driven analysis of sensitive changes in tractography may differentiate populations at risk for AD and isolate specific vulnerable networks.