Trees positively improve the annual thermal comfort of the built environment in tropical areas, where climate change is slight throughout the year. However, for areas with high changes in climate all year, the current studies have only explored the summer cooling performance of trees without the impact of different types of trees on annual thermal comfort, especially in cold seasons. Therefore, to quantify the impacts and scientifically guide the optimization of green space layout in hot summer and cold winter areas, this study selected Changsha City as the study area and analyzed how the annual thermal comfort is affected by evergreen trees and deciduous trees, which are two common types of trees in hot summer and cold winter areas. The analytical results indicated that the difference in the effect of deciduous and evergreen trees on outdoor thermal comfort was insignificant in summer, where the difference in the monthly mean PET for the three summer months was slight, being 0.28 °C, 0.14 °C, and 0.29 °C, respectively. However, evergreen trees greatly exacerbated winter cold compared to deciduous trees, with a monthly mean PET decrease by nearly 1.0 °C and an hourly PET reduced by up to 3.57 °C. The difference is mainly attributed to the absorption and reflection of solar radiation by the tree canopy, as well as the cooling and humidifying effect of the tree leaf. In hot summer and cold winter areas, outdoor thermal comfort is still in the “comfortable” and “slightly warm” acceptable stage despite the warming effect of deciduous trees in the spring and autumn seasons. Planting evergreen trees is an inevitable thermal mitigation choice for tropical areas. However, for the areas with high annual climate change, such as hot summer and cold winter areas in China, a change in empirical tree planting patterns and selecting deciduous trees where appropriate will improve year-round outdoor thermal comfort.