Black carbon (BC) in surface sediments from Henan section of Yellow River and Huaihe River, China, was determined. Average content of BC in Huaihe River was 0.33%, higher than that in Yellow River (n = 23) with mean value of 0.15%. Distribution patterns of BC in Yellow River and Huaihe River were similar, namely that tributaries had higher BC content than main stream. In addition, BC content in the mainstream of Yellow River and Huaihe River decreased with altitude. The BC content presented a significant positive correlation with clay (r = 0.672; p < 0.01) content in Yellow River, while neither did in Huaihe River. The ratio of BC/TOC ranged from 1.8–57.4% (median 29.6%), evidencing pyrogenic fossil fuel source of BC in Yellow River. Relatively low values of BC/TOC in Huaihe River (5.3%–28.8%, median 7.5%,) reflected that the origin of BC is from burning of biomass. In addition, Pearson rank correlation analysis showed that BC was in strong correlation with lighter PAH in Yellow River, while BC was in significant correlation with heavier PAH in Huaihe River. The ratio of BC/TOC indicated that BC in Yellow River mainly came from fossil fuel combustion, while BC in Huaihe River was primarily from biomass burning.