Abnormal behavior recognition using 3D-CNN combined with LSTM

Abstract

The research of abnormal behavior recognition is critical to personal and property security. In this paper, a 3D-CNN and Long Short-Term Memory (LSTM) based abnormal behavior recognition method has been proposed. The feature image composed of optical flow (OF) and motion history image (MHI) takes place of RGB image as the input of 3D-CNN. Because of the illumination changes and background jitter in complex scenes, a structural similarity background modeling method has been developed to suppress illumination variations. It is applied to updated dynamically both optical flow and motion history image. A new sample expansion method is developed to deal with the problem of abnormal behavior class imbalance. The OF and MHI feature image clips are randomly cropped firstly. Then clustering method is applied and cluster centers are collected to get new samples in quantity. LSTM with spatial temporal attention is developed to extract long-time spatial-temporal features for abnormal behavior recognition. Compared with state-of-the-art methods, our proposed method has excellent performance in abnormal behavior recognition on some challenging datasets.