Improving energy efficiency is an effective way to achieve carbon neutrality in the context of growing advocacy for a green economy in both academia and government. In this study, we analyzed the impact mechanism of the manufacturing agglomeration on total factor energy efficiency. Based on this work, we identified 30 provinces (autonomous regions and municipalities) in China using 2002 to 2017 panel data. We used the data envelopment analysis game cross-efficiency model to calculate the interprovincial current state of energy efficiency. We constructed a spatial Durbin model and used an adjacent space weight matrix, geographic distance weight matrix, and economic distance weight matrix to study the impact of manufacturing agglomeration on total factor energy efficiency. The results showed that under different spatial weights, the interprovincial total factor energy efficiency had a significant spatial dependence; under the three spatial weights, we identified a U-shaped relationship between manufacturing agglomeration and total factor energy efficiency. Industrial agglomeration had obvious spatial spillover effects on total factor energy efficiency, and the spillover effects under the weight of geographic distance were higher than other spatial weights. During the sample period, the direct, indirect, and total effects of the manufacturing industry had an impact on energy efficiency. It all had a negative number indicating that there was a crowding effect in manufacturing agglomeration, and it had an inhibitory effect on energy efficiency.