With ensuring food security becoming a priority for the Chinese government, the prevalence of off-farm employment (OE) may cast a shadow on agricultural productivity. Based on the data of the China Family Panel Studies in 2018, the Tobit model and threshold effect model have been applied to investigate the impact of off-farm employment on agricultural productivity efficiency (APE), measured by data envelopment analysis (DEA). The result has shown that: (1) OE contributes to a low level of APE. (2) Both self-employed off-farm employment (SOE) and wage-based off-farm employment (WOE) result in lower APE levels when endogenous issues are addressed. However, SOE had a greater negative impact on APE than WOE. (3) There exists a threshold for OE associated with a lower level of APE, indicating that the negative effect of OE on APE disappears when the degree of OE is high enough, SOE has a lower threshold than WOE. The study findings have implications for improving agricultural production efficiency in the context of large-scale off-farm employment of Chinese farmers.