Geo-environmental factors are believed to be major determinants of rural poverty. However, few studies have quantified the effects of these factors on rural poverty in China. In this paper, we used county-level poverty incidence data and geo-environmental factors to explore spatial patterns of the incidence of poverty using global and local spatial autocorrelation analysis and to investigate the effect of geo-environment factors on rural poverty using a geo-detector model. Our results demonstrated that there was spatial clustering of the incidence of poverty in the study area. The incidence of poverty decreased from south to north and from the east and west to the central area. The incidence of high–high poverty areas was mainly distributed in the southeast of Guizhou Province and the incidence of low–low poverty areas was distributed in the northeast. The results also demonstrated that percentage of effective irrigation on arable land, slope, elevation and vegetation cover were the dominant factors explaining the spatial pattern of poverty. Interaction analysis demonstrated that the slope non-linearly enhanced the percentage of effective irrigation on arable land. Our findings suggested that geo-environment is the fundamental control factor explaining the spatial pattern of rural poverty in China. Through analysis of the impact of the geo-environment on the spatial pattern of poverty, this study provides a reference for effectively implementing targeted alleviation of poverty.