Background: Hippocampal volume loss (HVL), PET-documented brain amyloid accumulation, and APOE-ε4 status are predictive biomarkers of the transition from mild cognitive impairment to Alzheimer disease (AD). In asymptomatic cases, the role of these biomarkers remains ambiguous. In contrast to the idea that HVL occurs in late phases of neurodegeneration, recent contributions indicate that it might occur before abnormal amyloid PET occurrence in elderly subjects and that its severity could be only marginally related to APOE variants. Using a longitudinal design, we examined the determinants of HVL in our sample, i.e., brain amyloid burden and the presence of APOE-ε4, and made a longitudinal assessment of cognitive functions. Methods: We performed a 4.5-year longitudinal study on 81 elderly community dwellers (all right-handed;, 48 (59.3%) women; mean age 73.7 ± 3.7 years) including MRI at baseline and follow-up, PET amyloid during follow-up, neuropsychological assessment at 18 and 54 months, and APOE genotyping. All cases were assessed using a continuous cognitive score (CCS) that took into account the global evolution of neuropsychological performance. Linear regression models were used to identify predictors of HVL. Results: There was a negative association between the CCS and HVL bilaterally. In multivariate models adjusting for demographic variables, the presence of APOE-ε4 was related to increased HVL bilaterally. A trend of significance was observed with respect to the impact of amyloid positivity on HVL in the left hemisphere. No significant interaction was found between amyloid positivity and the APOE-ε4 allele. Conclusion: The progressive decrement of neuropsychological performance is associated with HVL long before the emergence of clinically overt symptoms. In this cohort of healthy individuals, the presence of the APOE-ε4 allele was shown to be an independent predictor of worst hippocampal integrity in asymptomatic cases independently of amyloid positivity.