Summary China has been investing heavily in afforestation programmes to control soil erosion on the vast Chinese Loess Plateau (CLP). This massive afforestation has led to a considerable increase in forest and a decrease in dust‐storm activity in some areas. However, there have also been some negative outcomes, including low tree survival rate, increased soil erosion, exacerbated water shortages and deep soil desiccation. One important explanation for these is the use of inappropriate species because of a lack of knowledge of the natural vegetation in the area, which has been largely destroyed by human activities. Natural vegetation in the most recent warm period (the early–mid‐Holocene) can serve as an analogue for the ongoing greening programme, particularly under the global warming scenario. In this study, the natural vegetation of the CLP since the Last Glacial Maximum (LGM) was reconstructed from pollen analyses of six loess sections. Our results show that herbs were dominant both in the cold–dry LGM and the warm–humid early–mid‐Holocene. During the LGM, vegetation in the north‐western CLP mainly consisted of Artemisia, Echinops‐type, Taraxacum‐type and Chenopodiaceae, and vegetation in the south‐eastern CLP was characterized by the same types but with a slightly higher incidence of Poaceae. During the early–mid‐Holocene, vegetation was more diverse, with Poaceae, Artemisia, Echinops‐type and Chenopodiaceae dominant in the north‐west, and Pinus, Corylus, Poaceae, Artemisia and Selaginella sinensis dominant in the south‐east. Synthesis and applications. The ecological restoration of herbs should be considered a priority, although trees and shrubs have been prioritized previously. To balance environmental conservation and farm‐income support objectives, we suggest planting Corylus, Juglans and Selaginella sinensis in the south‐eastern Chinese Loess Plateau (CLP) because of their edible value or medicinal properties. Given the considerable prevalence of Selaginella sinensis and the Asteraceae family in the pollen records, and their useful medicinal effects, the CLP has great potential to be a centre for Chinese medicinal herb production.