The progression of Alzheimer's disease (AD) is characterized by complex trajectories of cerebral atrophy that are affected by interactions with age and apolipoprotein E allele ε4 (APOE4) status. In this article, we report the nonlinear volumetric changes in gray matter across the full biological spectrum of the disease, represented by the AD-cerebrospinal fluid (CSF) index. This index reflects the subject's level of pathology and position along the AD continuum. We also evaluated the associated impact of the APOE4 genotype. The atrophy pattern associated with the AD-CSF index was highly symmetrical and corresponded with the typical AD signature. Medial temporal structures showed different atrophy dynamics along the progression of the disease. The bilateral parahippocampal cortices and a parietotemporal region extending from the middle temporal to the supramarginal gyrus presented an initial increase in volume which later reverted. Similarly, a portion of the precuneus presented a rather linear inverse association with the AD-CSF index whereas some other clusters did not show significant atrophy until index values corresponded to positive CSF tau values. APOE4 carriers showed steeper hippocampal volume reductions with AD progression. Overall, the reported atrophy patterns are in close agreement with those mentioned in previous findings. However, the detected nonlinearities suggest that there may be different pathological processes taking place at specific moments during AD progression and reveal the impact of the APOE4 allele.