Objectives: Surveillance colonoscopy is recommended after polypectomy because adenoma recurrence is common. The aim of this study was to evaluate the predictors of high-risk adenoma occurrence at surveillance colonoscopy in patients who undergo colorectal adenoma removal and to clarify the association between age and recurrent colorectal adenoma. Methods: This retrospective study included 563 patients who had colorectal adenomas at baseline colonoscopy and who underwent surveillance colonoscopy. The risk factors for recurrent adenoma were evaluated and the 5-year cumulative incidence rates of overall and high-risk adenoma were compared according to age group. Results: During a mean follow-up period of 3.1 years, 305 (54.2%) patients had overall adenoma recurrence, and 80 (14.2%) patients had high-risk adenoma at surveillance colonoscopy. In a multivariate analysis, old age (≥60 years) and presence of multiple adenomas (3 or more) were significantly associated with high-risk adenoma (p = 0.002 and p = 0.006 respectively). The 5-year cumulative incidence rates of high-risk adenoma were 7.4, 16.7, and 24.1% in the < 50, 50–59, and ≥60 years group respectively (p < 0.001). Conclusions: Old age (≥60 years) and presence of multiple adenomas (3 or more) were strongly associated with the occurrence of high-risk adenoma at surveillance colonoscopy. The 5-year cumulative incidence of high-grade adenoma was significantly high in the old age group.