Objective The aim of this study was to review existing evidence on the efficiency and safety of Chinese herbal medicine for the treatment of prediabetes. Methods Randomized controlled trials (RCTs) of Chinese herbal medicine (CHM) to treat prediabetes were searched in the following databases from their inception date onwards until 2 May 2020: MEDLINE, Cochrane, EMBASE, Web of Science, EBSCO, CINAHL, CNKI, VIP database, CBM, and Wanfang database. Quality assessment of included trials was accessed according to the guidance in Cochrane. Researchers independently assessed the validity of included trials and extracted outcome data for synthesis. RevMan 5.3 was used for the meta-analysis. Results Twenty-two RCTs including 3923 participants were included in the study. Our findings upon the 22 RCTs showed CHM is effective in the treatment of prediabetes, which can statistically reduce the incidence of diabetes (RR = 0.48; 95% CI = (0.41, 0.57); P < 0.001), increase the incidence of normalization of prediabetes (RR = 1.76; 95% CI = (1.57, 1.96); P < 0.001), and lower FPG (MD = −0.38; 95% CI = (−0.60, −0.16); P < 0.001), 2hPG (MD = −1.13; 95% CI = (−1.60, −0.67); P < 0.001), TG (MD = −0.23; 95% CI = (−0.33, −0.13); P < 0.001), TC (MD = −0.34; 95% CI = (−0.52, −0.16); P < 0.001), and BMI (MD = −0.48; 95% CI = (−0.78, −0.18); P < 0.001) after treatment, and there was no difference of HbA1c (P > 0.05). Conclusion CHM is effective for the treatment of prediabetes. CHM can statistically reduce the incidence of diabetes, increase the incidence of normalization of prediabetes, and lower the FPG, 2hPG, TG, TC, and BMI levels, but with no significant difference in HbA1c. In addition, CHM was relatively safe in clinical practice. More high-quality RCTs should be conducted to strengthen the finding.