It is poorly known how Aβ and tau accumulations associate at the spatiotemporal level in the in vivo human brain to impact cognitive changes in older adults prior to AD symptoms onset. In this study, we used a graph theory-based spatiotemporal analysis to characterize the cortical patterns of Aβ and tau deposits and their relationship with cognitive changes in the Harvard Aging Brain Study (HABS) cohort. We found that the temporal accumulations of interlinked Aβ and tau pathology display distinctive spatiotemporal correlations associated with early cognitive decline. Notably, we observed that baseline Aβ deposits-Thal amyloid phase Ⅱ-related to future increase of tau deposits, Braak stages Ⅰ-Ⅳ, both displaying linkage to the decline in multi-domain cognitive scores. We also found unimodal tau-to-tau and cognitive impairment associations in broad areas of Braak stages Ⅰ-Ⅳ. The unimodal Aβ-to-Aβ progressions were not associated with cognitive changes. Our results revealed a multifaceted correlation of the spatiotemporal Aβ and tau associations with cognitive decline over time, in which tau-to-tau and tau-Aβ interactions, and not Aβ independently, might be critical contributors to clinical trajectories toward AD in older adults.