GssMILP for anomaly classification in surveillance videos

Abstract

Multiple Instance Learning (MIL) is widely used to locate abnormal video frames in surveillance videos due to its ability to work with weakly-labeled data. On the other hand, graph-based semi-supervised approaches are employed to handle label sparsity-related issues such as Low Degree-of-Supervision (LDoS) and High Class-Imbalance (HCI). However, the application of the MIL paradigm in anomaly classification of surveillance videos faces significant challenges such as (i) LDoS, (ii) HCI, and (iii) A multitude of types of abnormal events and actions instead of a unique pattern. The current work proposes a novel anomaly classifier to address these issues with a new objective function (GssMILP) by leveraging the benefits from graph-based semi-supervised and multiple-instance learning approaches. In this regard, our contributions are six-fold. First, a deep hierarchical architecture (C3D+3DCNN BICLSTM) to extract generic video descriptors. These descriptors represent the multitude of types of abnormal events and actions by incorporating the local–global spatial information and long-short-term temporal information related to the object motion, human actions, and behavioral features. Second, we develop a learning objective (GssMILP) that extends the MIL-based instance-level labeler to graph-based semi-supervising. Third, we use three different regularizer terms, namely graph-based regularizer, non-convex L2-regularizer, and temporal-smoothness regularizer. Fourth, we present an elaborate experimental study comparing the performance of the proposed approach with six similar baseline approaches. The results indicate that at degree-of-supervision 1.0%, the proposed method improves the f1-score by 45.26%, 18.19%, 45.28%, and 9.22% in UCSD-ped1, UCSD-ped2, MED, and RWAD datasets, respectively. Fifth, we present an ablation study on the impact of different features on the performance of an anomaly classifier. Sixth, we provide temporal annotations for a label resource of 1900 videos in the RWAD dataset.