From the perspective of input and output differentiation, using the SBM directional distance function method and Luenberger productivity index, this paper calculates the index of the green total factor productivity (GTFP) growth of 280 cities in China, from 2004 to 2016. This study also analyzes the growth sources of urban GTFP and the decomposition situation of GTFP in four cases, as well as the factors that affect the growth of GTFP. Finally, the following conclusions are drawn: 1) the overall growth trend of GTFP under four different situations of elements input is basically consistent. As a whole, the growth trend can be divided into three stages. The first rising stage of GTFP growth was from 2004 to 2008; the next declining stage was from 2008 to 2012, and a further rising stage occurred from 2012 to 2016. 2) From the decomposition situation of GTFP growth, we can know that the main source of GTFP growth is in the scale efficiency (LSEC) of GTFP. The decomposition of inefficiency levels in four cases shows that the inefficiency of land resources is the highest among many factors. Land resource inefficiency is also the key factor determining the output efficiency; the input inefficiency and bad-output inefficiency between them account for about 50% of the overall inefficiency level. 3) Among the factors that influence GTFP growth, the negative effect of urban land and energy structure is mainly reflected in the pure technology growth (LPTP) and scale efficiency growth (LSEC) of GTFP. In addition, investment in scientific and technological innovation and foreign direct investment both have a significantly positive effect on GTFP growth, as well as in improving urban population density and road area per capita.