Large-scale revegetation in desertified land might promote the sequestration of global soil carbon, but the influencing factors on inorganic carbon accumulation processes that absorb CO2 through abiotic “inorganic respiration” processes remain unclear. Several study sites were selected in the Mu Us Sandy Land: quicksand land (0 years), semi-fixed sand land (6 years), and three nearby shrub and arbor forest lands (20, 30, and 51 years). During 20–51 years of revegetation, soil organic carbon density (SOCD) and inorganic carbon density (SICD) significantly increased by 2.5–7.5 times and 1.7–7.0 times for shrub forest and 3.1–7.5 times and 2.6–7.8 times for arbor forest compared with quicksand land, but the difference between the two studied forest lands were not statistically significant. After revegetation in desertified land, the content of total nitrogen (TN), available potassium (AK), cation exchange capacity (CEC), clay, Ca2+, and Mg2+ also greatly enhanced, and they were positively correlated with SOC and SIC, while bulk density and pH slightly decreased, which were negatively correlated with SOC and SIC. T-value biplot importance ranking further indicated that TN was the most sensitive factor affecting the accumulation of SOC and SIC, and its contribution rate to SOC and SIC reached 89.8% and 88.1%, respectively. Moreover, SOC was also related to AK, CEC, and clay, while SIC was more affected by clay, Ca2+, and Mg2+ during revegetation. In conclusion, afforestation with shrubs and arbors on the Mu Us Sandy Land promoted the sequestration of SOC and SIC synergistically by modified soil physicochemical properties.