Rapid urbanization is an important factor leading to the rise in surface temperature. How to effectively reduce the land surface temperature (LST) has become a significant proposition of city planning. For the exploration of LST and the urban heat island (UHI) effect in Zhengzhou, China, the LST was divided into seven grades, and the main driving factors of LST change and their internal relations were discussed by correlation analysis and gray correlation analysis. The results indicated that LST showed an upward trend from 2005 to 2020, and a mutation occurred in 2013. Compared with 2005, the mean value of LST in 2020 increased by 0.92 °C, while the percentage of LST-enhanced areas was 22.77. Furthermore, the spatial pattern of UHI was irregularly distributed, gradually spreading from north to south from 2005 to 2020; it showed a large block distribution in the main city and southeast in 2020, while, in the areas where woodlands were concentrated and in the Yellow River Basin, there was an obvious “cold island” effect. In addition, trend analysis and gray correlation analysis revealed that human factors were positively correlated with LST, which intensified the formation of the UHI effect, and the influence of Albedo on LST showed obvious spatial heterogeneity, while the cooling effect of vegetation water was better than that of topography. The research results can deepen the understanding of the driving mechanism of the UHI effect, as well as provide scientific support for improving the quality of the urban human settlement environment.