Rs3851179, a variant of PICALM gene, and age are the risk factors of Alzheimer's disease (AD). AD is divided into early-onset AD (EOAD, < 65 years) and late-onset AD (LOAD, ≥ 65 years) by age. The purpose was to investigate the impact of different genotypes of PICALM rs3851179 on brain atrophy and cognitive decline across the AD continuum in different age groups. Four hundred seven cognitive normal (CN) controls, 362 mild cognitive impairment (MCI) patients, and 94 AD patients were enrolled to assess the interaction between AD continuum, age status, and PICALM on gray matter volume (GMV), global cognition, memory function, and executive function using full factorial ANCOVA (3 × 2 × 2). The interaction between AD continuum and PICALM significantly affected the GMV of the left putamen (PUT.L). rs3851179 A-allele carriers did not show a significant decrease in PUT.L GMV from CN to MCI to AD, while GG-allele carriers did. The interaction between AD continuum and age status was significant on GMV of the left angular gyrus (ANG.L) and right superior occipital gyrus (SOG.R). LOAD had higher GMV of ANG.L and SOG.R than EOAD. The interactive effects among AD continuum, age status, and PICALM were not significant on GMV but were significant on global cognition and executive function. The A-allele was found to have a protective effect on global cognition and executive function in EOAD, but not significantly so in LOAD. PICALM rs3851179 A-allele might alleviate the atrophy of PUT.L across the AD continuum than GG-allele. Age status did not affect the interaction between AD continuum and PICALM on brain atrophy. The ANG.L and SOG.R atrophied more severely in EOAD than in LOAD. Rs3851179 A-allele was protective for global cognition and executive function in EOAD.