Tau neurofibrillary tangles have a central role in the pathogenesis of Alzheimer’s Disease (AD). Mounting evidence indicates that the propagation of tau is assisted by brain connectivity with weakened white-matter integrity along the propagation pathways. Recent advances in tau positron emission tomography tracers and diffusion magnetic resonance imaging allow the visualization of tau pathology and white-matter connectivity of the brain in vivo. The current study aims to investigate how tau deposition and structural connectivity are associated with memory function in prodromal AD. In this study, tau accumulation and structural connectivity data from 83 individuals (57 cognitively normal participants and 26 participants with mild cognitive impairment) were associated with neurocognitive test scores. Statistical analyses were performed in 70 cortical/subcortical brain regions to determine: 1. the level of association between tau and network metrics extracted from structural connectivity and 2. the association patterns of brain memory function with tau accumulation and network metrics. The results showed that tau accumulation and network metrics were correlated in early tau deposition regions. Furthermore, tau accumulation was associated with worse performance in almost all neurocognitive tests performance evaluated in the study. In comparison, decreased network connectivity was associated with declines in the delayed memory recall in Craft Stories and Benson Figure Copy. Interaction analysis indicates that tau deposition and dysconnectivity have a synergistic effect on the delayed Benson Figure Recall. Overall, our findings indicate that both tau deposition and structural dysconnectivity are associated with neurocognitive dysfunction. They also suggest that tau-PET may have better sensitivity to neurocognitive performance than diffusion MRI-derived measures of white-matter connectivity.