Aiming at the high complexity of existing crowd abnormal detection models, the inability of traditional CNN to extract time-related features, and the lack of training samples, an improved spatial-temporal convolution neural network is proposed in this paper. The algorithm firstly uses the aggregation channel feature model to process the surveillance image, and selects the suspected object region with saliency characteristics. Then, the scaled correction and feature extraction are performed on the obtained suspected object region. The corresponding low-level features are obtained and input into the deep network for deep feature learning so as to enhance the representation ability. Finally, the deep feature is input into the least squares SVM classification model to obtain the final abnormal behavior detection result. The embedded chip Hi353I is used as the hardware processor to realize the real-time abnormal behavior detection effect. Our proposed deep intelligent analysis algorithm is used as abnormal Behavior Detector in the board level test. The results show that most of abnormal behaviors can be detected and the alarming message can be timely transmitted in the real-time surveillance.