Rural livelihood resilience (RLR) is significant for the sustainability of rural areas, farmers and agriculture. This research takes the mountainous areas of Southwest China as the study areas and integrates the buffer, adjustment and renewal capabilities to construct a RLR analysis framework and evaluation indicator system. The RLR of 234 sample counties was evaluated using set pair analysis, and the influencing factors of RLR were investigated using Geodetector. The aim is to scientifically analyze the vertical spatial differentiation and influencing factors of RLR and fully explore the risk-resistant potential of rural livelihood systems in mountainous areas. The results show that (1) From 2000 to 2020, RLR increases significantly, but the level is still low, and its structure shows a primary pattern of “buffer capability > adjustment capability > renewal capability”. (2) There is no significant negative correlation between RLR and terrain gradients, especially in the middle and low mountains. RLR in high mountainous areas is significantly lower than in low and medium mountainous areas, but the gap is narrowing. There are no significant gaps in RLR and various capabilities between low and medium mountain areas. (3) Economic development, non-agricultural industries and grassroots autonomous organizations are essential determinants for RLR. Economic development, grassroots autonomous organizations and transportation conditions are more important in low and medium mountain areas, while non-agricultural industries and medical conditions contribute more to high mountainous areas. (4) To enhance the RLR, policy recommendations should place a strong emphasis on extending the agricultural industry chain, improving rural production and living infrastructure and strengthening the supply of high-quality social public services. The findings can provide a scientific basis for governments to implement rural revitalization strategies and improve farmers’ well-being, as well as practical guidance for enhancing the risk resistance ability of underdeveloped mountainous rural areas.