Agricultural labor productivity reflects the relationship between labor factor input and economy in the agricultural industry, and improving agricultural labor productivity is an important way to achieve agricultural modernization and high-quality agricultural development. Based on the VAR model and Granger causality test, this paper conducts an empirical analysis of the impact of urban-rural income gap, urbanization, industrial structure upgrading, and agricultural labor productivity in China from 1981 to 2020, and concludes that there is a long-term cointegration relationship between urban and rural income gap, urbanization and industrial structure upgrading and agricultural labor productivity. In the long run, the urban-rural income gap and industrial structure upgrading have a positive impact on agricultural labor productivity. Urbanization has a negative impact on the improvement of agricultural labor productivity, so relevant suggestions are given.