The traditional detection of impurities in wheat has difficulties such as low precision, time-consuming, and cumbersome, therefore, it is important to study the method of rapid and accurate detection of impurities in wheat for correctly assessing the quality grade of wheat. Terahertz (THz) technology has many superior properties such as transient, broadband, low-energy, and penetrating, which can realize rapid and nondestructive detection of wheat quality. In this study, a classification and recognition algorithm AHA-RetinaNet-X for wheat impurity terahertz images based on RetinaNet and Artificial hummingbird algorithm (AHA) is proposed.A THz three-dimensional tomography imaging system is used to image wheat and its impurities, and two THz image datasets, respectively the wheat and impurity dataset for verifying the classification effect of wheat and impurities and the impurity dataset for verifying the classification effect of impurities. The experimental results show that the AHA-RetinaNet-X model outperforms other detection and classification models in terms of accuracy, F1-score, precision, recall, and specificity, and is able to achieve 96.1%, 94.9%, 95.2%, 95.8%, 95.5%, 95.3%, and 93.3% for the wheat and impurity dataset and the impurity dataset, respectively, 95.6%, 96.3%, and 95.2%, and the mAP value of AHA-RetinaNet-X is also higher than the other models and can reach 92.1%. The combination of THz imaging technology and AHA-RetinaNet-X can realize the classification and identification of wheat and impurities, which provides a new method for the non-contact rapid nondestructive detection and identification of wheat and impurities, and also provides a reference for the research of the identification and detection methods of other substances.