Herbs have been used as natural remedies for disease treatment, prevention, and health care. Some herbs with functional properties are also used as food or food additives for culinary purposes. The quality and safety inspection of herbs are influenced by various factors, which need to be assessed in each operation across the whole process of herb production. Traditional analysis methods are time-consuming and laborious, without quick response, which limits industry development and digital detection. Considering the efficiency and accuracy, faster, cheaper, and more environment-friendly techniques are highly needed to complement or replace the conventional chemical analysis methods. Infrared (IR) and Raman spectroscopy techniques have been applied to the quality control and safety inspection of herbs during the last several decades. In this paper, we generalize the current application using IR and Raman spectroscopy techniques across the whole process, from raw materials to patent herbal products. The challenges and remarks were proposed in the end, which serve as references for improving herb detection based on IR and Raman spectroscopy techniques. Meanwhile, make a path to driving intelligence and automation of herb products factories.