Improving agricultural energy inefficiency is essential for achieving sustainable agricultural development and promoting major agricultural countries to achieve carbon peak and carbon neutrality goals. This paper analyzes agricultural energy inefficiency in China, using panel data from 30 provinces between 2000 and 2021. The by-production technology model is employed to measure and decompose inefficiency, and the simultaneous equations model and moderating effect model are utilized to study the impact mechanism of industrial agglomeration, land consolidation, and agricultural energy inefficiency. The findings reveal several key points: First, the average inefficiency of agricultural energy in China increased from 0.370 to 0.514, with economic inefficiency rising at a faster rate than environmental inefficiency. Second, agricultural industrial agglomeration serves to inhibit both agricultural energy economic inefficiency and environmental inefficiency, which, in turn, hampers the development of industrial agglomeration. This relationship shows heterogeneity across the eastern, central, and western regions, as well as between major and non-major grain production areas. Third, land consolidation—both nationally and specifically in the central, major grain-producing, and non-major grain-producing areas—effectively mitigates the deterioration of agricultural energy inefficiency caused by industrial agglomeration. In the eastern region, land consolidation can enhance the inhibitory effect of industrial agglomeration on energy inefficiency. This paper highlights the interconnections between industrial agglomeration, land consolidation, and agricultural energy inefficiency, providing valuable policy references for the development of sustainable agriculture and the proactive and steady advancement of carbon peak and carbon neutrality goals.