The growth of pathogens potentially relevant to respiratory tract infection may be triggered by changes in ambient temperature. Few studies have examined the association between ambient temperature and pneumonia incidence, and no studies have focused on the susceptible elderly population. We aimed to examine the short-term association between ambient temperature and geriatric pneumonia and to assess the disease burden attributable to cold and hot temperatures in Hong Kong, China. Daily time-series data on emergency hospital admissions for geriatric pneumonia, mean temperature, relative humidity, and air pollution concentrations between January 2005 and December 2012 were collected. Distributed-lag nonlinear modeling integrated in quasi-Poisson regression was used to examine the exposure-lag-response relationship between temperature and pneumonia hospitalization. Measures of the risk attributable to nonoptimal temperature were calculated to summarize the disease burden. Subgroup analyses were conducted to examine the sex difference. We observed significant nonlinear and delayed associations of both cold and hot temperatures with pneumonia in the elderly, with cold temperatures having stronger effect estimates. Among the 10.7% of temperature-related pneumonia hospitalizations, 8.7% and 2.0% were attributed to cold and hot temperatures, respectively. Most of the temperature-related burden for pneumonia hospitalizations in Hong Kong was attributable to cold temperatures, and elderly men had greater susceptibility.