Soil organic carbon (SOC) plays a vital role in improving soil quality and alleviating global warming. Understanding the dynamic changes in SOC is crucial for its accumulation induced by compost application in agroecosystem. In this study, soil samples were collected from three treatments: high-rate bio-compost (BioMh), low-rate bio-compost (BioMl), and control (CK, no fertilization) during 2002–2020 in a wheat-maize double cropping system in North China. The soils were separated into three functional fractions, i.e., coarse particle organic matter (cPOM, >250 μm), microaggregates (μAgg, 53–250 μm) and mineral-associated organic matter (MAOM, < 53 μm), and the associated SOC contents were determined. During 1993–2002, SOC contents in bulk soil significantly increased with the duration in the BioMh and BioMl plots. However, there was no significant correlation between SOC content and duration during 2002–2020. These results suggested that compost application positively improved SOC sequestration, while the duration of SOC sequestration (i.e., the longevity of increased SOC with time) under compost inputs maintained only 9 years. Moreover, there was a significant increase in mean annual SOC contents in bulk soil with compost application rate during 2002–2020, indicating that carbon saturation did not occur. Additionally, the SOC contents in the cPOM fraction increased with time (p < 0.01), but the corresponding μAgg and MAOM associated SOC was insignificant (p > 0.05). The MAOM fraction exhibited no additional carbon accumulation with expanding compost application, confirming a hierarchical carbon saturation in these fractions. We concluded that soils under wheat-maize double cropping system in North China have greater potential to sequester C through additional compost inputs, despite showing hierarchical saturation behavior in the non-protected coarse particulate fraction.