Improving agricultural energy efficiency is essential in reducing energy consumption and achieving agricultural sustainable development. This paper aims to measure the agricultural total-factor energy efficiency in China rather than the partial-factor energy efficiency while taking full account of regional heterogeneity and to investigate the driving factors of agricultural total-factor energy efficiency. The empirical results showed that the average value of agricultural total-factor energy efficiency is 0.814 in China, and the technological gap ratio is 0.853. The regional difference in agricultural total-factor energy efficiency was quite obvious. Higher agricultural energy inputs are associated with higher agricultural total-factor productivity. The total value of potential agricultural energy savings in 30 provinces of China reached 1704.41 billion tons of standard coal. In terms of the absolute amount of agricultural energy saving, the amount was largest in the low-energy-input area, which was 113.87 million tons of standard coal, accounting for 66.81% of the total potential saving amount. Furthermore, we used the Tobit model to analyze the influencing factors of agricultural total-factor energy efficiency. We found that the proportion of agriculture to GDP has a positive impact on agricultural total-factor energy efficiency, while the per capita income of farmers, fiscal support for agriculture, the illiteracy rate of farmers, agricultural labor input, and agricultural capital stock have a negative impact on agricultural total-factor energy efficiency. Finally, we proposed policy implications in terms of agricultural technological progress, agricultural infrastructure, technical training, etc.