Abstract This paper constructs a 5G network security detection system based on the functional requirements of network security detection and the P2DR process model. The structure can be broken down into three layers from the bottom to the top: acquisition layer, analysis layer, and display layer. The design focuses on writing the SVM algorithm into the software of the communication network security vulnerability monitoring system, and in order to solve the defect of the long training time of the model of this machine learning algorithm, the incremental learning vector machine model is used, which is combined into the CSV-KKT-ISVM model. Test datasets that cover system performance, effectiveness, and leakage are used to test the system after it is completed. The test data was analyzed to prove that the system’s memory usage was maintained at 46M, CPU usage was 5% to 10%, and the response time was no later than 1 s. The monitoring accuracy was 98.5% at the highest but decreased with the increase of the percentage of vulnerability data, and the accuracy dropped to 93.9% at 50%, the minimum was not lower than 90%, and the error rate was no less than 0.8%. To achieve the best outcome, the system threshold should be set to 5, and there should be no false alarms or misreporting.