The goal of carbon neutralization in 2060 in China have stressed the importance of increasing energy efficiency in energy-intensive industries, while the factor market distortion has seriously hindered the effective allocation of resources. Focusing on the metallurgical industry, this paper seeks to explore the influence of eliminating factor price distortion on energy efficiency, which could not only provide elaborate recommendations in this crucial sector but also have theoretical implications for building an efficient energy system in the future. To fulfill this goal, we first measure the relative price distortion of production factors in the metallurgical industry, and then explore the impact of dispelling the relative price distortion on total factor energy efficiency on the basis of inter-factor substitution elasticity. The results show that the relative price distortion among production factors exists in China's metallurgical industry, and the prices of labor and energy are relatively higher than that of capital. Eliminating factors price distortion would bring an 18.8% growth in energy efficiency in China's metallurgical industry. Hence, the process of market-oriented reform in the factor market should be accelerated to help construct an efficient energy system in China.