Tau deposition is one of two hallmark features of biologically defined Alzheimer's disease (AD) and is more closely related to cognitive decline than amyloidosis. Further, not all amyloid-positive individuals develop tauopathy, resulting in wide heterogeneity in clinical outcomes across the population with AD. We hypothesized that a polygenic risk score (PRS) based on tau PET (tau PRS) would capture the aggregate inherited susceptibility/resistance architecture influencing tau accumulation, beyond solely the measurement of amyloid-β burden. Leveraging rich multimodal data from a population-based sample of older adults, we found that this novel tau PRS was a strong surrogate of tau PET deposition and captured a significant proportion of the variance in tau PET levels as compared with amyloid PET burden, APOE (apolipoprotein E) ε4 (the most common risk allele for AD), and a non-APOE PRS of clinical case-control AD risk variants. In independent validation samples, the tau PRS was associated with cerebrospinal fluid phosphorylated tau levels in one cohort and with postmortem Braak neurofibrillary tangle stage in another. We also observed an association of the tau PRS with longitudinal cognitive trajectories, including a statistical interaction of the tau PRS with amyloid burden on cognitive decline. Although additional study is warranted, these findings demonstrate the potential utility of a tau PRS for capturing the collective genetic background influencing tau deposition in the general population. In the future, a tau PRS could be leveraged for cost-effective screening and risk stratification to guide trial enrollment and clinical interventions in AD.