While China's economy is developing rapidly, the problem of carbon emissions is indeed more prominent. The development of green finance is conducive to supporting environmental improvement and enhancing carbon productivity. Most of the existing literature examines the relationship between green innovation and carbon productivity from the perspectives of industrial structure, technological innovation, and economic growth. However, the mechanism by which green finance (GF) affects carbon productivity and whether there is heterogeneity remains unclear. Therefore, this study utilizes panel data from 277 cities from 2010 to 2020 and employs a mediation effect model to investigate the relationship between GF and carbon productivity. The results of the study found that GF has a positive U-shaped association with carbon productivity. The quantity and the quality of green innovation have a mediating effect on the above relationship. It may be due to the fact that GF promotes the green transformation and sustainable development of the economic structure by supporting green industry and technological innovation, which provides strong support to enhance carbon productivity. In non-resource cities, the impact of GF on carbon productivity is more obvious through improving the quality of green innovation, probably because non-resource cities have the advantages of a diversified industrial structure, stronger innovation capacity, and easier access to policy support and market mechanism support than resource cities. The quantity and the quality of green innovation in the eastern, central, and western regions all play a mediating role. The findings provide policymakers with recommendations for utilizing GF in a two-carbon environment to achieve sustainable low-carbon development.