Physical exercise is recognized for its beneficial effects on brain health and executive function, particularly through the careful manipulation of key exercise parameters, including type, intensity, and duration. The aim of this systematic review and meta-analysis was to delineate the optimal types, intensities, and durations of exercise that improve cognitive functions in older adults with mild cognitive impairment (MCI) or dementia. A comprehensive search was conducted in Scopus, Web of Science, and PubMed from their inception until December 2023. The methodological quality and publication bias of the included studies were assessed using the PEDro scale and Egger's regression test, respectively. Separate meta-analyses were performed to assess the overall impact of exercise on cognitive assessments and to explore the effects of different exercise types (i.e., aerobic, resistance, dual-task, mind-body, and multi-component exercises) and intensities (i.e., low, moderate, and high) on executive function. Results were presented as standardized mean differences (SMD) and 95% confidence intervals (95% CI). A meta-regression analysis was conducted to examine the correlation between exercise duration and mean effects. In total, 15,087 articles were retrieved from three databases, of which 35 studies were included in our final analyses. The results indicated high overall methodological quality (PEDro score = 8) but a potential for publication bias (t = 2.08, p = 0.045). Meta-analyses revealed that all types of exercise (SMD = 0.691, CI [0.498 to 0.885], p < 0.001) and intensities (SMD = 0.694, CI [0.485 to 0.903], p < 0.001) show significant effects favoring exercise. Notably, dual-task exercises (SMD = 1.136, CI [0.236 to 2.035], p < 0.001) and moderate-intensity exercises (SMD = 0.876, CI [0.533 to 1.219], p < 0.001) exhibited the greatest effect. No significant correlation was observed between exercise duration and SMD (R² = 0.038, p = 0.313). Overall, our meta-analyses support the role of physical exercise in enhancing executive function in older adults with MCI or dementia. It is essential to carefully tailor exercise parameters, particularly type and intensity, to meet the specific needs of older adults with MCI or dementia. Such customization is crucial for optimizing executive function outcomes and improving overall brain health.