Large amounts of carbon in alpine sediments have been expected to be sensitive to climate change, but how carbon accumulation responds to climate change remains unclear. Thus, we explored the impact of different factors on the carbon accumulation rate (CAR) of alpine sediments by combining a variety of climatic variables, vegetation data and erosion indicators based on two alpine sediment successions on Taibai Mountain, the highest peak in central and eastern mainland China. One succession is near the modern treeline (Paomaliang Swamp, PML) and the other is located at the upper forest line (Sanqing Chi, SQC, a small lake). We used our previously published organic carbon content data and for the first time calculated the CAR, and further used pollen and physicochemical indicators to quantify the contributions from climate, vegetation and soil. We found that their contributions varied during different periods and between the two sediment successions. For the PML succession, from 5850 to 4000 calendar years before present (cal. a BP), the CAR was low, which was related to low annual temperatures, low vegetation cover and strong soil erosion. From 4000 to 2400 cal. a BP, a high CAR coincided with high annual temperatures, high vegetation cover and weak soil erosion. From 2400 to 200 cal. a BP, the CAR decreased, mainly attributed to low vegetation cover. Local vegetation cover had major impacts on the CAR in the SQC succession during the Middle–Late Holocene. In general, the local factor interpretation rate in SQC (83%) was higher than that of PML (47%), related to the vegetation stability of continuous forest and the treeline. This study highlights the important role of the local environment in determining carbon accumulation in the alpine region.