Vegetation phenology is a commonly used indicator of ecosystem responses to climate change and plays a vital role in ecosystem carbon and hydrological cycles. Previous studies have mostly focused on the response of vegetation phenology to temperature and precipitation. Soil moisture plays an important role in maintaining vegetation growth. However, our understanding of the influences of soil moisture dynamics on vegetation phenology is sparse. In this study, using a time series of the normalized difference vegetation index (NDVI) from the moderate resolution imaging spectroradiometer (MODIS) dataset (2001–2020), the start of the growing season (SOS), the end of the growing season (EOS), and the length of the growing season (LOS) in the Qilian Mountains (QLMs) were extracted. The spatiotemporal patterns of vegetation phenology (SOS, EOS, and LOS) were explored. The partial coefficient correlations between the SOS, EOS, and seasonal climatic factors (temperature, precipitation, and soil moisture) were analyzed. The results showed that the variation trends of vegetation phenology were not significant (p > 0.05) from 2001 to 2020, the SOS was advanced by 0.510 d/year, the EOS was delayed by 0.066 d/year, and the LOS was prolonged by 0.580 d/year. The EOS was significantly advanced and the LOS significantly shortened with increasing altitude. The seasonal temperature, precipitation, and soil moisture had spatiotemporal heterogeneous effects on the vegetation phenology. Overall, compared with temperature and soil moisture, precipitation had a weaker influence on the vegetation phenology in the QLMs. For different elevation zones, the temperature and soil moisture influenced the vegetation phenology in most areas of the QLMs, and spring temperature was the key driving factor influencing SOS; the autumn soil moisture and autumn temperature made the largest contributions to the variations in EOS at lower (<3500 m a.s.l.) and higher elevations (>3500 m a.s.l.), respectively. For different vegetation types, the spring temperature was the main factor influencing the SOS for broadleaf forests, needleleaf forests, shrublands, and meadows because of the relative lower soil moisture stress. The autumn soil moisture was the main factor influencing EOS for deserts because of the strong soil moisture stress. Our results demonstrate that the soil moisture strongly influences vegetation phenology, especially at lower elevations and water-limited areas. This study provides a scientific basis for better understanding the response of vegetation phenology to climate change in arid mountainous areas and suggests that the variation in soil moisture should be considered in future studies on the influence of climate warming and environmental effects on the phenology of water-limited areas.