During the long preclinical phase of dementia, accelerated cognitive impairment is regarded as a cardinal marker. Thus, the identification of risk factors for cognitive impairment is of great significance for dementia prevention. This study aims to examine the joint associations of sleep duration and physical activity with cognitive impairment among rural elderly over 65 years old, and provide suggestions for improving the cognitive function in rural elderly over 65 years old. A cross-sectional study was conducted in rural Nanjing by recruiting 1147 individuals aged above 65 years. Cognitive function was assessed using the brief community screening instrument for dementia. Physical activity was assessed using the Global Physical Activity Questionnaire. Data were analyzed by multivariate logistic regression models, and a significant difference was set at P < 0.05. Compared with participants with proper sleep duration and sufficient physical activity, participants with short sleep duration and insufficient physical activity (odds ratio (OR): 1.820; 95% CI: 1.265 ~ 2.618), long sleep duration and sufficient physical activity (OR: 2.428; 95% CI: 1.137 ~ 5.183) showed an increased likelihood of cognitive impairment. Inappropriate sleep duration combined with insufficient physical activity was associated with a significantly higher likelihood of cognitive impairment in rural elderly over 65 years old.