Abstract In recent years, the way of air travel is gradually popularized, and the security check system in the airport is facing corresponding pressure. In order to guarantee the security problems in the aviation field, this study investigates the security check system in the civil aviation field. After investigating the security checking process in civil aviation, the Faster R-CNN has been improved based on both multilayer feature fusion and loss function to adapt to the target detection of millimeter wave images. Then, the detection and radiation characteristics of millimeter waves are introduced, and the target detection algorithm based on edge extraction is constructed. On this basis, artificial intelligence, big data, augmented reality glasses, and other technologies are incorporated to establish a security check system in civil aviation and analyze it experimentally. In terms of the overall recognition performance of mAP, this paper’s method improves by an average of 3.098-6.504% and 3.740-8.706%, respectively, with the highest FPS of 45 and 36, and its generalization ability is better compared to other methods. In addition, in the practice of contraband detection in unmixed and mixed backgrounds, the devices based on the security system of this paper have higher than 90% detection in all seven types of contraband, which is able to improve the accuracy of contraband detection, while still maintaining good performance in the case of overlapping targets. This study can assist security personnel in completing the task of detecting contraband and improving the efficiency of security checks.