To investigate the association between current and former smoking and the risk of mortality in elderly Chinese men. Our study participants were elderly (≥ 60 years) men recruited in a suburban town of Shanghai. Cigarette smoking status was categorized as never smoking, remote (cessation > 5 years) and recent former smoking (cessation ≤ 5 years), and light-to-moderate (≤ 20 cigarettes/day) and heavy current smoking (> 20 cigarettes/day). Cox proportional hazards models and restricted cubic splines were used to examine the associations of interest. The 1568 participants had a mean age of 68.6 ± 7.1 years. Of all participants, 311 were never smokers, 201 were remote former smokers, 133 were recent former smokers, 783 were light-to-moderate current smokers and 140 were heavy current smokers. During a median follow-up of 7.9 years, all-cause, cardiovascular and non-cardiovascular deaths occurred in 267, 106 and 161 participants, respectively. Heavy current smokers had the highest risk of all-cause and non-cardiovascular mortality, with an adjusted hazard ratio (HR) of 2.30 (95% CI: 1.34-4.07) and 3.98 (95% CI: 2.03-7.83) versus never smokers, respectively. Recent former smokers also had a higher risk of all-cause (HR = 1.62, 95% CI: 1.04-2.52) and non-cardiovascular mortality (HR = 2.40, 95% CI: 1.32-4.37) than never smokers. Cox regression restricted cubic spline models showed the highest risk of all-cause and non-cardiovascular mortality within 5 years of smoking cessation and decline thereafter. Further subgroup analyses showed interaction between smoking status and pulse rate (≥ 70 beats/min vs. < 70 beats/min) in relation to the risk of all-cause and non-cardiovascular mortality, with a higher risk in current versus never smokers in those participants with a pulse rate below 70 beats/min. Cigarette smoking in elderly Chinese confers significant risks of mortality, especially when recent former smoking is considered together with current smoking.