Relocating poor people in mountainous areas to centralized settlement sites is an important poverty alleviation policy implemented by China and a key measure to promote stable poverty alleviation and sustainable rural development for the poor. Based on the survey data of 405 peasant households in the Panxi Area in 2021, this study constructed a structural equation model (SEM) to explore the influencing factors of peasant households’ willingness to relocate to a concentrated residence. The results showed the following: (1) Of the 405 peasant households surveyed, 20.98% were unwilling to move to centralized resettlement sites, making it more difficult to carry out the relocation policy for poverty alleviation. (2) Living environment, living conditions, important social groups, the economic benefits, living benefits, and survival benefits brought by concentrated residences, governments, and the village committees significantly influenced the willingness to relocate to a concentrated residence. In contrast, agricultural income, ecological benefits, and value benefits brought by the concentrated residence had little effect on the willingness to relocate to a concentrated residence. (3) Land force, human force, cognitive force, and national force significantly positively affected the peasant households’ willingness to relocate to a concentrated residence. This study is of great significance in promoting the implementation of poverty alleviation and relocation policy, improving the efficiency of relocation and promoting the wellbeing of peasant households.