The incorporation of plant-derived biomass by microorganisms into deceased microbial biomass, i.e., a “microbial carbon pump”, is essential for forming a soil carbon (C) pool. Therefore, microbial communities and associated functions could shape the formation of soil organic C (SOC) composition and persistence. However, the mechanism by which microorganisms mediate the degradation of various types of biomass (such as plant or microbial) and its impact on SOC formation under different vegetation types remains unclear. Herein, we analyzed microbial communities, carbohydrate-activated enzymes (CAZymes), absolute quantification of C-degradation genes, and extracellular enzyme activities to track microbial-mediated SOC formation under three vegetation types on the Loess Plateau. Changes in vegetation type influenced microbial community structure and distributions of certain bacterial groups. Specifically, dominant bacterial taxa shifted from oligotrophic Actinobacteria to eutrophic Proteobacteria from grass to forest soil. The proportion of microbial CAZymes responsible for decomposing plant-derived components (86–89%) exceeded that of microbial-derived components (11–14%), suggesting a greater capacity for the degradation of deceased plant biomass by microorganisms. There was a reduction of 14%–17% in the absolute abundance of specific C-degradation genes for hemicellulose and cellulose in forest soil compared to grass, while an increase of 20%–32% was observed in lignin and chitin degradation. This indicated a higher decomposition potential of lignin and chitin by microorganisms in forest soil. The number of CAZymes genes involved in the degradation of bacteria-derived biomass (peptidoglycan) was higher than fungi-derived biomass (chitin and glucans) and had a close correlation with the qCO2, microbial biomass C (MBC), and particulate organic C (POC). The abundance of C-degrading genes increased with the increase in the corresponding enzyme activity, indicating that enzyme activity is regulated by functional genes. In conclusion, shifts in CAZymes genes encoding for the degradation of diverse carbon sources could impact microbial metabolic activity through a microbial “carbon pump” regulation process. This mechanism could facilitate the formation of soil organic carbon and its fractions across different vegetation types.