Water erosion exerts a profound, but uncertain effect on the terrestrial carbon cycle. Therefore, the soil erosion intensity is uncertain in different terrains. Previous studies have focused on the slope and intensity of soil erosion, while ignored the special terrain such as gully created by erosion. To advance the understanding of the mineralization dynamics of soil organic carbon (SOC) from the micro perspective, the relationship between soil erosion and carbon source and sink is expected to be further improved. In this study, we explored the impact of water erosion on SOC mineralization in the areas of gully erosion in Qingyuan mountain, Jiangxi, China by combining the 137Cs technique and laboratory incubation experiment, calculated the first-order kinetic model parameters of the SOC mineralization of bulk soil and aggregates in the area of gully erosion, and estimated the correlation between the factors that may affect SOC mineralization. The results showed that (1) the mineralization rates of SOC in the bulk soil and aggregates showed fluctuations and reached a peak, and finally became stable. The soil erosion intensity was significantly positively correlated (p < 0.05) with the cumulative mineralization of SOC. (2) The cumulative mineralization in the depositional area (489.67 mg kg−1) was higher than that in the erosional area (402.69 mg kg−1). (3) Soil macroaggregates were the main contributors to SOC mineralization (63.56%–92.28%), and the soil macroaggregate organic carbon mineralization dynamics was more sensitive to soil erosion. Additionally, significant effects (p < 0.05) of soil mechanical composition, capillary pores, saturated water content, and contents of carbon and nitrogen on SOC mineralization were observed. Erosion increased the risk of SOC loss by mineralization of SOC, especially in macroaggregates. The deposition process caused the enrichment and burial of SOC, and the background level of SOC and the content of macroaggregates in the depositional area were higher than those in the erosional area. The results showed that soil erosion increased the risk of SOC loss by mineralization of SOC based on laboratory simulation. The present study aiming at examining the effect of gully erosion on the mineralization and sequestration of SOC provides important insights into balancing the global carbon budget.