Drylands are terrestrial ecosystems sensitive to climate change. There are totally drylands of 5.17 × 106 km2 (above 80% of global total temperate desert area) in Central Asia (CAS), in which significant increases of temperature and changes of precipitation have been detected in recent decades. However, environment-induced changes in terrestrial carbon stocks of these dryland ecosystems have not been well investigated. With the Arid Ecosystem Model (AEM), this study was devoted to analyze spatiotemporal changes of carbon stocks in drylands over CAS during the past 35 years (1980–2014) and to quantify contributions to these changes of various factors, including temperature, precipitation, and atmospheric CO2 concentration. Over the study period, total stocks of vegetation carbon (VEGC), soil organic carbon (SOC), and litter carbon (LTRC) averaged 2.8 ± 0.05 Pg C, 45.2 ± 0.01 Pg C, and 0.3 ± 0.004 Pg C(1Pg = 1015 g) in CAS, respectively. Meanwhile, total carbon (TOTC) declined by 0.7 Pg C. Climate change caused TOTC to decrease by 1.3 Pg C. In contrast, CO2 enrichment effect caused TOTC to increase by 0.9 Pg C. The effects of different factors on TOTC changes varied spatially. Precipitation was the dominant factor regulating TOTC change in 40.9% of the study area, mainly in the desert sparse shrub region in northwest Kazakhstan and the dryland region of southern Xinjiang of China, in which vegetation growth was mainly limited by water resource. CO2 dominated the change of TOTC in 38.3% of the study area, mainly in the lower altitude regions of Tianshan mountain, in which the hydrothermal condition was relatively suitable for vegetation growth. Ecosystems in southern Xinjiang of China and northwest Kazakhstan are fragile to climate change.