Ecological protection and high-quality development of the Yellow River Basin (YRB), China, aroused remarkable concerns from China’s Central Government, and has been a major national strategy. The Inner Mongolia reach of the Yellow River Basin (IM-YRB) is a typical dryland with pervasive vegetation restoration through the actions of the ecological projects that have been conducted in recent years. However, how climate changes and human activities, such as land use and land cover (LULC) changes, jointly impact vegetation variations in this region remains poorly understood. Here, using an explainable machine learning technique, we evaluated linkages between the kernel normalized difference vegetation index (kNDVI) and air temperature, precipitation, soil moisture, and LULC changes, and relevant marginal contributions of these four drivers to the observed vegetation changes. The grassland fraction on a pixel level was selected as the quantitative LULC variable due to its key role in regional LULC. We found that interannual kNDVI changes in most areas of this study region were negatively sensitive to temperature, but positively sensitive to precipitation and soil moisture, with area fractions of 71.74%, 96.93%, and 89.33%, respectively. The area fraction of negative kNDVI sensitivity to LULC was roughly equivalent to that of positive kNDVI sensitivity. The contributions of air temperature, precipitation, soil moisture, and LULC to overall kNDVI changes were 21.54%, 33.32%, 32.19%, and 12.95%, respectively. Moisture conditions also play a critical role in vegetation changes, which was reflected by the fluctuating growth of kNDVI as interannual changes in precipitation. Nonetheless, kNDVI changes are also affected by LULC, and LULC became the dominant factor behind the kNDVI anomalies over the grassland restoration regions from barren over the IM-YRB. This research provides theoretical support for dryland vegetation restoration under the influence of climate change.