Southwestern China (SW) is simultaneously affected by the East Asian monsoon, South Asian monsoon and westerly winds, forming a complex and diverse distribution pattern of climate types, resulting in a low interpretation rate of vegetation changes by climate factors in the region. This study explored the response characteristics of vegetation to climatic factors in the whole SW and the core area of typical climate type and the phased changes in response, adopting the form of “top-down”, using linear trend method, moving average method and correlation coefficient, and based on the climate data of CRU TS v. 4.02 for the period 1982–2017 and the annual maximum, 3/4 quantile, median, 1/4 quantile, minimum and average (abbreviated as P100, P75, P50, P25, P5 and Mean) of GIMMS NDVI, which were to characterize vegetation growth conditions. Coupling with the trend and variability of climate change, we identified four major types of climate change in the SW, including the significant increase in both temperature and precipitation (T+*-P+*), the only significant increase in temperature and decrease (T+*-P−) or increase (T+*-P+) of precipitation and no significant change (NSC). We then screened out nine typical areas of climate change types (i.e., core areas (CAs)), followed by one T+*-P+* area, which was located in the center of the lake basin of the Qiangtang Plateau. The response of vegetation to climatic factors in T+*-P+* area/T+*-P+ areas and T+*-P− areas/NSC areas were mainly manifested in an increase and a significant decrease, which makes the response characteristics of vegetation to climatic factors in the whole SW have different directionality at different growth stages. Our results may provide new ideas for clearly showing the complexity and heterogeneity of the vegetation response to climate change in the region under the background of global warming.