Climate change and human activities have considerably changed the spatial patterns and functional elements of regional habitats. Understanding spatiotemporal changes in habitat quality (HQ) and their potential driving factors is essential for maintaining ecosystem health and protecting biodiversity. To explore the effect of physical and human factors on HQ changes in Southwest China, we firstly analyzed the land-use change intensity (LCI). We then evaluated spatiotemporal changes in HQ based on the InVEST model and explored the spatial heterogeneity of the main driving factors of HQ changes based on a geographical detector and a geographical weighted regression model. The results showed that LCI had obvious spatiotemporal differences, and LCI from low-quality habitat to high-quality habitat (LCI1) was significantly higher than that from high-quality habitat to low-quality habitat (LCI2). The HQ improved steadily in Southwest China in 1990–2015, showing a trend of low–high–low from southeast to northwest. Moreover, there were twelve factors, including aboveground biomass, ecological land area ratio, population density, slope, etc., which had a significant impact on the spatial differences in HQ, and the effects of different factors on HQ had observable spatial heterogeneity. The effect of LCI2 on the spatial difference of HQ was greater than that of LCI1. These results suggested that the current ecosystem protection and management policy had a positive effect on improving HQ. Our study provides an important decision-making reference for sustainable land development and utilization and regional ecological protection and restoration.