Local residents of buffer zones, as a key factor in the World Heritage conservation and sustainable development, have not received sufficient attention in most developing countries, especially in the mountainous areas where poor and backward ethnic minorities live. To fill this research gap, this paper takes the Karst World Heritage buffer zone in Libo, Guizhou Province, southwest mountainous area of China, as the research area, and explores the factors that influence the perception of residents’ responsibility for the World Heritage conservation by taking local residents who are involved in tourism management as the research subjects. Data were collected in the buffer zone of the Libo World Heritage site and 186 valid data were generated. SPSS 25.0 and AMOS 27.0 software were used to analyze the questionnaire data and construct a structural equation model. The results showed that environmental protection behavior had the greatest impact on residents’ perception of responsibility for heritage conservation (0.93), followed by the recognition of heritage value (0.55), tourism positive impact (0.39), and place identification (0.34), among which the positive impact of tourism had a greater impact on the perception of heritage value (0.52). The results of the study emphasize the importance of the recognition of heritage value and positive tourism influence on the formation of residents’ perception of responsibility for heritage conservation, and provide an empirical basis for the conservation of the World Natural Heritage.