Video surveillance system is widely used in modern society. Human behavior recognition based on neural network uses computer vision and image processing technology to analyze and judge the behavior of different objects in surveillance video. In order to improve the recognition accuracy of human behavior, this paper makes an in-depth study on the design of video surveillance system and target tracking using the relevant algorithms of neural network. In this paper, the application of relevant neural network algorithms and the technology of character behavior recognition are analyzed by means of experimental method comparison and algorithm comparison. The experimental data shows that in the detection of abnormal behavior, the highest accuracy rate of human behavior recognition is 90% by using the improved hu invariant moment model.