The COVID-19 pandemic is a health issue leading older adults to an increased vulnerability to unfavorable outcomes. Indeed, the presence of physical frailty has recently led to higher mortality due to SARS-CoV-2 infection. However, no longitudinal studies have investigated the role of neuropsychogeriatric factors associated with lockdown fatigue in healthy cognitive aging. Eighty-one healthy older adults were evaluated for their neuropsychological characteristics, including physical frailty, before the pandemic (T0). Subsequently, 50 of them agreed to be interviewed and neuropsychologically re-assessed during the lockdown (T1) and immediately after it (T2). Moreover, during another home confinement, they performed a psychological screening (T3) to evaluate possible mood changes and fatigue. According to Fried's frailty criteria, at T0, 63% of the sample was robust, 34.5% pre-frail, and only 2.5% frail. Significantly, these subjects presented a decrease in handgrip strength and walking speed (29.6 and 6.1%, respectively). Results from Principal Component Analyses and multiple regression models highlighted the contribution of “cognitive” and “psychological” factors (i.e., attentive-executive performance and mood deflections) in explaining handgrip strength and gait speed. At T3, lockdown fatigue was explained by higher scores on the Beck Depression Inventory and lower scores on the Trail Making Test part A. Results from a moderated-mediation model showed that the effect of psychomotor speed on lockdown fatigue was mediated by depression, with a moderating effect of gait speed. Our findings highlight the complex interrelationship between cognitive, psychological, and physical factors in the emergence of pandemic fatigue in a carefully selected older population.