Modifiable lifestyle-related factors such as smoking and alcohol drinking are associated with cognitive impairment in the elderly population but the relationships have shown various results. To evaluate the relationship of alcohol drinking and smoking in the early 60 s with the risk of developing incident cognitive impairment. In 1999, we evaluated cognitive function, smoking, and drinking status in 3,174 inhabitants aged 60–64 years in a rural area of Korea, with a follow-up assessment of cognitive function 7 years later. A total of 1,810 individuals who did not show cognitive impairment at baseline were included. A stratified analysis was applied to evaluate how smoking and alcohol drinking affected the risk of developing cognitive impairment based on gender. Current smokers showed a higher risk for developing cognitive impairment than did never smokers (odds ratio [OR], 1.53; 95% confidence interval [CI], 1.09–2.15). The OR for female current smokers compared with never smokers was 1.62 (95% CI, 1.05–2.52), and smokers with higher pack-years were more likely to develop cognitive impairment than never smokers, showing a dose–response relationship (P for trend = 0.004). Frequent alcohol consumption increased the risk of developing cognitive impairment (OR, 1.68; 95% CI, 1.01–2.78), and a dose–response relationship was observed among male subjects (P for trend = 0.044). Infrequent drinking in females decreased the odds of developing cognitive impairment (OR, 0.67; 95% CI, 0.42–1.00), whereas frequent drinking tended to increase the odds, although this trend was not significant, suggesting a U-shaped relationship. Although the sample was small for some analyses, especially in female, our data suggest that smoking and drinking in the early 60 s are associated with a risk of developing cognitive impairment, and this relationship is characterized by gender differences.