Functional ability decline occurs with age. This study aims to investigate the associations between the lifestyle factors-exercising, food consumption, and smoking-and the functional ability of the activities of daily living (ADL) by gender. The data were obtained from the Taiwan longitudinal study on aging, a national cohort study. The cross-sectional results demonstrated that the frequency of exercising was negatively associated with ADL in both men and women. Dairy products were positively associated with ADL in men. The longitudinal results illustrated that current and consistent exercising were negatively associated with changes in ADL scores over 4- and 8-year periods in both men and women. Seafood consumption was negatively associated with changes in the subsequent 4-year ADL scores. Past smoking was positively associated with changes in subsequent 4-year ADL scores in men, while current smoking was positively associated with changes in subsequent 8-year ADL scores in women. Therefore, consistent exercising, food consumption, and smoking were associated with ADL functional ability in elderly people, and the associations differed by gender. Elders exercising consistently had good ADL performance and maintained their ADL ability better, especially women. Seafood consumption decreased the risk of ADL decline, while smoking increased the risk of ADL decline.