The Chinese Altai is one of the most typical segments of the Altai-Mongolia terrane, of which the tectonic evolution and affinity are hotly disputed and still not well constrained. Early Paleozoic metasedimentary rocks are extensive in the Altai-Mongolia terrane and their provenance is a key to unravelling the tectonic history of this terrane. Metasediments from Kulumuti Group were collected from the central Chinese Altai for geochemical study. They have low Chemical Index of Alteration (CIA = 52–54) and high Index of Compositional Variability (ICV = 0.81–1.19) values, different from the mature post-Archean Australian average shale (PAAS), indicating relatively weak chemical weathering and a source compositionally dominated by immature material that lacks alumina-rich minerals. These rocks are moderately enriched in light rare earth element (LREE) and show relatively flat heavy rare earth element (HREE) patterns (LaN/YbN = 2.36–9.80, GdN/YbN = 1.31–2.45). Compared with PAAS, they mostly have lower large ion lithophile element concentrations (e.g. Rb, Sr, Ba, Th, U and Pb) and similar contents of high field strength elements (e.g. Zr, Hf and Y), but with lower Nb and Ta. The metasediments have high Rb concentrations (> 50 ppm), relatively high K2O (> 1.1 wt.%) and low REE contents with negative Eu/Eu* anomalies, indicating that these rocks were derived from an acid-intermediate igneous source. The rocks have slightly enriched Nd isotopic compositions with calculated initial εNd(t) values mostly ranging from − 4.3 to − 0.2 and TDM2 model ages between 1.22 and 1.56 Ga. Detrital zircons from two samples of the Kulumuti metasediments have similar age spectra, dominated by Early Paleozoic to latest Neoproterozoic zircons (465–576 Ma) with a few older grains formed between 766–972 Ma and 1321–2572 Ma, remarkably consistent with those from the low-grade metamorphic Habahe Group in the western Chinese Altai. In combination with previous published data, the detrital zircons from Early Paleozoic metasediments in the Chinese Altai dominantly formed between 430–580 and 750–950 Ma, with only a minor population of Meso- to Paleoproterozoic grains. When compared with adjacent terranes, the age spectrum of the Chinese Altai is most similar to that of the Tarim Craton. This craton provided Precambrian materials for the Early Paleozoic metasedimentary rocks now exposed in the Chinese Altai, and therefore suggests that the whole Altai-Mongolia terrane has a close tectonic affinity to the Tarim Craton.