Overuse of chemical fertilizer in apple production has resulted in environmental contamination in China. Existing research related to fertilizer use efficiency has focused on grain crops, and few studies have considered cash crops despite the significant structural shift of fertilizer application from grain to cash crops and the impacts of climate factors on fertilizer use efficiency. This paper employed a stochastic frontier method to examine the technical efficiency and fertilizer use efficiency of apple production by adopting panel data of eight major apple production provinces in China from 1992 to 2014. Further, the panel random-effects Tobit model was used to explore the factors influencing fertilizer use efficiency. The fertilizer use efficiency score ranged within 0.002–0.878, with a mean value of 0.472, which was much lower and had greater variability than technical efficiency with an average value of 0.868. Irrigation fees, land size, non-agricultural employment opportunities, apple planting area ratio and fertilizer prices had significant positive effects on fertilizer use efficiency in apple production. Disaster ratio, precipitation, sunshine hours, and apple prices had a significant negative effect on fertilizer use efficiency. Formula fertilizer promoted by governments had no significant effect on improving fertilizer use efficiency. Fertilizer use efficiencies are more sensitive to education level and fertilizer price than non-agricultural employment opportunity and land size, and it is weakly sensitive to climate factors, apple price, and apple planting area ratio. Policies aimed at improving fertilizer use efficiency and reducing environment pollution should, therefore, focus on promoting non-agricultural employment opportunities and land circulation, strengthening and expanding rural extension services, raising environmental protection awareness and controlling fertilizer price, and encouraging farmers to adopt new production technology.