Surveillance Video Summarization Research Articles

Deep Learning Assists Surveillance Experts: Toward Video Data Prioritization

Video summarization (VS) suppresses high-dimensional (HD) video data by only extracting only the important information. However, prior research has not focused on the need for surveillance VS, that is used for many applications to assist video surveillance experts, including video retrieval and data storage. In addition, mainstream techniques commonly use 2D deep models for VS, ignoring event occurrences. Accordingly, we present a two-fold 3D deep learning-assisted VS framework. First, we employ an inflated 3D ConvNet model to extract temporal features; these features are optimized using a proposed encoder mechanism. The input video is temporally segmented using a feature comparison technique for selecting a single frame from each video segment. The segmented shots are evaluated using our novel shot segmentation evaluation scheme and are input into a saliency computation mechanism for keyframe selection in a second fold. Qualitative and quantitative analyses over VS benchmarks and surveillance videos demonstrate the superior performance of our framework, with 0.3- and 4.2-unit increases in the F1 scores for YouTube and TVSum datasets, respectively. Along with accurate VS, a key contribution of our study is the novel shot segmentation criterion prior to VS, which can be used as a benchmark in future research to effectively prioritize HD visual data.

Crowd aware summarization of surveillance videos by deep reinforcement learning

Surveillance videos which record crowd behaviors have dramatically increased due to the wide applications. A quick view of such crowd surveillance video in a constrained time is an increasing demand because it always contain a huge number of redundancy frames. In this paper, we focus on summarization of crowd surveillance videos. But it is not easy due to two reasons. First, how to make the decision to keep or discard a subshot from the input surveillance video stream so that the summary can outline the main behaviors of the crowd over a limited frames sequence. Second, how to maintain performance of summarization model for long surveillance videos. To tackle these challenges, we formulate surveillance video summarization as a sequential decision-making process and train the summarization network with reinforcement learning-based framework. A novel crowd location-density reward is proposed to teach summarization network to produce high-quality summaries. In addition, a summarization network with three layers LSTM is designed to maintain performance across longer time spans. Extensive experiments on three public crowd surveillance videos datasets show that the proposed method achieves state-of-the-art performance.

Cost-Effective Video Summarization Using Deep CNN With Hierarchical Weighted Fusion for IoT Surveillance Networks

Video summarization (VS) has attracted intense attention recently due to its enormous applications in various computer vision domains, such as video retrieval, indexing, and browsing. Traditional VS researches mostly target at the effectiveness of the VS algorithms by introducing the high quality of features and clusters for selecting representative visual elements. Due to the increased density of vision sensors network, there is a tradeoff between the processing time of the VS methods with reasonable and representative quality of the generated summaries. It is a challenging task to generate a video summary of significant importance while fulfilling the needs of Internet of Things (IoT) surveillance networks with constrained resources. This article addresses this problem by proposing a new computationally effective solution through designing a deep CNN framework with hierarchical weighted fusion for the summarization of surveillance videos captured in IoT settings. The first stage of our framework designs discriminative rich features extracted from deep CNNs for shot segmentation. Then, we employ image memorability predicted from a fine-tuned CNN model in the framework, along with aesthetic and entropy features to maintain the interestingness and diversity of the summary. Third, a hierarchical weighted fusion mechanism is proposed to produce an aggregated score for the effective computation of the extracted features. Finally, an attention curve is constituted using the aggregated score for deciding outstanding keyframes for the final video summary. Experiments are conducted using benchmark data sets for validating the importance and effectiveness of our framework, which outperforms the other state-of-the-art schemes.

Fog computing enabled cost-effective distributed summarization of surveillance videos for smart cities

Fog computing is emerging an attractive paradigm for both academics and industry alike. Fog computing holds potential for new breeds of services and user experience. However, Fog computing is still nascent and requires strong groundwork to adopt as practically feasible, cost-effective, efficient and easily deployable alternate to currently ubiquitous cloud. Fog computing promises to introduce cloud-like services on local network while reducing the cost. In this paper, we present a novel resource efficient framework for distributed video summarization over a multi-region fog computing paradigm. The nodes of the Fog network is based on resource constrained device Raspberry Pi. Surveillance videos are distributed on different nodes and a summary is generated over the Fog network, which is periodically pushed to the cloud to reduce bandwidth consumption. Different realistic workload in the form of a surveillance videos are used to evaluate the proposed system. Experimental results suggest that even by using an extremely limited resource, single board computer, the proposed framework has very little overhead with good scalability over off-the-shelf costly cloud solutions, validating its effectiveness for IoT-assisted smart cities.

Efficient CNN based summarization of surveillance videos for resource-constrained devices

The widespread usage of surveillance cameras in smart cities has resulted in a gigantic volume of video data whose indexing, retrieval and management is a challenging issue. Video summarization tends to detect important visual data from the surveillance stream and can help in efficient indexing and retrieval of required data from huge surveillance datasets. In this research article, we propose an efficient convolutional neural network based summarization method for surveillance videos of resource-constrained devices. Shot segmentation is considered as a backbone of video summarization methods and it affects the overall quality of the generated summary. Thus, we propose an effective shot segmentation method using deep features. Furthermore, our framework maintains the interestingness of the generated summary using image memorability and entropy. Within each shot, the frame with highest memorability and entropy score is considered as a keyframe. The proposed method is evaluated on two benchmark video datasets and the results are encouraging compared to state-of-the-art video summarization methods.

Key-frame selection for automatic summarization of surveillance videos: a method of multiple change-point detection

Recent years have witnessed a drastic growth of various videos in real-life scenarios, and thus there is an increasing demand for a quick view of such videos in a constrained amount of time. In this paper, we focus on automatic summarization of surveillance videos and present a new key-frame selection method for this task. We first introduce a dissimilarity measure based on f-divergence by a symmetric strategy for multiple change-point detection and then use it to segment a given video sequence into a set of non-overlapping clips. Key frames are extracted from the resulting video clips by a typical clustering procedure for final video summary. Through experiments on a wide range of testing data, excellent performances, outperforming given state-of-the-art competitors, have been demonstrated which suggests good potentials of the proposed method in real-world applications.

Adaptive fusion of human visual sensitive features for surveillance video summarization.

Surveillance video cameras capture large amounts of continuous video streams every day. To analyze or investigate any significant events, it is a laborious and boring job to identify these events from the huge video data if it is done manually. Existing approaches sometimes neglect key frames with significant visual contents and/or select some unimportant frames with low/no activity. To solve this problem, in this paper, a video summarization technique is proposed by combining three multimodal human visual sensitive features, such as foreground objects, motion information, and visual saliency. In a video stream, foreground objects are one of the most important pieces of a video as they contain more detailed information and play a major role in important events. Moreover, motion is another stimulus of a video that significantly attracts human visual attention. To obtain this, motion information is calculated in the spatial domain as well as the frequency domain. Spatial motion information can select object motion accurately; however, it is sensitive to illumination changes. On the other hand, frequency motion information is robust to illumination change, although it is easily affected by noise. Therefore, motion information in both the spatial and the frequency domains is employed. Furthermore, the visual attention cue is a sensitive feature to measure the indication of a user's attraction label for determining key frames. As these features individually cannot perform very well, they are combined to obtain better results. For this purpose, an adaptive linear weighted fusion scheme is proposed to combine the features to rank video frames for summarization. Experimental results reveal that the proposed method outperforms the state-of-the-art methods.

Retrospective analysis of time series for frame selection in surveillance video summarization

In recent days, we have witnessed a dramatical growth of videos in various real-life scenarios. In this paper, we address the problem of surveillance video summarization. We present a new method of key-frame selection for this task: By virtue of retrospective analysis of time series, temporal cuts are first detected by sequentially measuring dissimilarities on a given video with threshold-based decision making; then, with the detected cuts, the video is segmented into a number of consecutive clips containing similar video contents; key frames are last selected by performing a typical clustering procedure in each resulted clip for final video summary. We have conducted extensive experiments on the benchmarking ViSOR dataset and the publicly available IVY LAB dataset. Excellent performances outperforming state-of-the-art competitors were demonstrated on key-frame selection for surveillance video summarization, which suggests good potentials of the proposed method in real-world applications.

Unsupervised, efficient and scalable key-frame selection for automatic summarization of surveillance videos

Recent years have witnessed a dramatical growth of the deployment of vision-based surveillance in public spaces. Automatic summarization of surveillance videos (ASOSV) is hence becoming more and more desirable in many real-world applications. For this purpose, a novel frame-selection framework is proposed in the present paper, which has three properties: 1) un-supervision: it can work without requirements of any supervised learning or training; 2) efficiency: it can work very fast, with experiments demonstrating efficiency faster than real-timeness and 3) scalability: it can achieve a hierarchical analysis/overview of video content. The performance of proposed framework is systematically evaluated and compared with various state-of-the-art frame selection techniques on some collected video sequences and publicly-available ViSOR dataset. The experimental results demonstrate promising performance and good applicability for real-world problems.

Event-based large scale surveillance video summarization

Recent advances in sensor manufacture and computer vision technologies have simulated the applications of intelligent transportation systems, while a key yet under-addressed issue in these systems is the semantic summarization of large scale surveillance video. The main difficulty of large scale surveillance video summarization arises from the contradiction between the high-degree spatiotemporal redundancies and the limited storage budget. In this paper, we propose a novel approach of large scale surveillance video summarization on the basis of event detection. In the proposed approach, we firstly obtain the trajectories of vehicles and pedestrians in a tracking-by-detection manner, and then detect the abnormal events using the trajectories. Finally, we design a disjoint max-coverage algorithm to generate a summarized sequence with maximum coverage of interested events and minimum number of frames. Compared with traditional key frame-based approaches, our approach enjoys the following favorable features. First, important information can be efficiently extracted from the redundant contents since the approach is event-centric and those interested events contain almost all the important information. Second, abnormal events are successfully detected by combining the Random Forest classifier and the trajectory features. Third, the abnormal events are designed to display, and hence further reduces the compression ratio. Due to the above features, the proposed approach is suitable for different scenarios, ranges from highway to crowded crossings. Experiments on 12 surveillance sequences validate the effectiveness and efficiency of the proposed approach.

Eye-gaze based real-time surveillance video synopsis

This paper presents a novel real-time surveillance video summarization system that employs the eye-gaze positions of the surveillance operator. The system output can be used to efficiently review the overlooked sections of the surveillance video, which increases the reliability of the surveillance system. The summary of the operator-monitored actions can also be obtained for efficient re-examination of the surveillance videos. The system employs a novel non-linear video abstraction method that can mix actions from different frames into the same summary frame for more compact videos. The video summaries are performed in real-time on average hardware thanks to our improved dynamic programming based summary techniques. We performed several experiments using synthetic and real world surveillance videos, which showed the practical applicability of our system. Sample videos and their summaries can be reached at http://vision.gyte.edu.tr/projects.php?id=5.