Agriculture is the source of human clothing and food, but it also brings negative externalities to the environment. The outflow of the rural population is one of the factors for changes in the characteristics of the rural population. Farmers' decisions on agricultural production can affect agricultural ecological efficiency. Therefore, it is necessary to study the relationship between the two in rural development. Taking Qin'an County in the Loess Hilly Region of central Gansu, China, as an example, this paper analyzed the demographic characteristics and the evolution characteristics of agricultural eco-efficiency under the background of rural population outflowing, and the impact of the former on the latter, based on the panel statistical data of 17 villages and towns from 2001 to 2020. The results show that (1) From 2001 to 2020, the non-agricultural level of Qin'an County's labor force showed an upward fluctuation trend. The level of aging was relatively stable, and the per capita disposable income was significantly increased. (2) From 2001 to 2020, the agricultural eco-efficiency of Qin'an County showed a wavy change, but there were some towns and villages that have not been effectively developed. The regional differences are significantly different. (3) The non-agriculturalization level of the labor force promotes agricultural eco-efficiency through the direct effect rather than the space spillover effect. The positive effect of aging on agricultural eco-efficiency was mainly reflected through direct effect rather than spatial spillover effect. Per capita, disposable income has a significant positive spatial spillover effect on agricultural eco-efficiency. Finally, this paper provides a scientific reference for promoting the improvement of agricultural eco-efficiency and sustainable development. This is of great theoretical and practical significance for the realization of rural revitalization.