Video-based Surveillance Systems Research Articles

A maneuver indicator and ensemble learning-based risky driver recognition approach for highway merging areas

Abstract Due to the complex traffic characteristics in highway merging areas, drivers tend to exhibit high-risk driving behaviors. To address the characteristics of driving behavior in highway merging areas, we have developed a real-time identification model for risky drivers by combining a driver risk level labeling method with load balancing-ensemble learning (LB-EL). In this paper, we explore four types of maneuver indicator indexes (MIIs)—acute direction, stomp pedal, dangerous following, and dangerous lane changing—that can describe the negative behaviors of both individual vehicles and vehicle platoons in highway merging areas. To quantize the label driver risk level, we use the interquartile range (IQR) method and Criteria Importance Though Intercriteria Correlation (CRITIC), while we evaluate the reliability of the MII using spatial analysis. Furthermore, we balance the dataset using three load balancing (LB) algorithms and create nine ensemble strategies by pairing adaptive boosting (AdaBoost), extreme gradient boosting (XGBoost), and light gradient boosting machine (LGBM) with the three LB algorithms. Finally, we validate the proposed model using trajectory data extracted from UAV videos. The results indicate that the distribution laws of risky driving behaviors in the acute direction and stomp pedal show a high degree of similarity and good matching with the distribution laws of traffic conflict points in existing research. Moreover, the SMOTE-LGBM ensemble model achieves the best performance, reaching an accuracy rate of 93.4% and a recall rate of 92.1%, which demonstrates the validity of our proposed model. This model can be widely applied to recognize risky drivers in video-based surveillance systems.

Online Video Anomaly Detection.

With the popularity of video surveillance technology, people are paying more and more attention to how to detect abnormal states or events in videos in time. Therefore, real-time, automatic and accurate detection of abnormal events has become the main goal of video-based surveillance systems. To achieve this goal, many researchers have conducted in-depth research on online video anomaly detection. This paper presents the background of the research in this field and briefly explains the research methods of offline video anomaly detection. Then, we sort out and classify the research methods of online video anomaly detection and expound on the basic ideas and characteristics of each method. In addition, we summarize the datasets commonly used in online video anomaly detection and compare and analyze the performance of the current mainstream algorithms according to the evaluation criteria of each dataset. Finally, we summarize the future trends in the field of online video anomaly detection.

Automated Video-Based Air Traffic Surveillance System for Counting General Aviation Aircraft Operations at Non-Towered Airports

The vast majority of U.S. airports are not equipped with control towers, which limits their ability to keep records of flight operations. Attempts have been made to use sensor-based technologies to count aircraft operations at non-towered airports; however, they exhibit limited accuracy. To this end, we developed an automated video-based surveillance system capable of detecting general aviation aircraft departure and landing operations, which comprise the vast majority of operations at non-towered airports. The proposed computer vision method is comprised of three modules: aircraft detection, aircraft tracking, and operations count and classification. We explored different camera layouts and state-of-the-art machine learning and deep learning methods to determine the best settings to extract operations trajectory features for operations count and classification. The proposed method was tested at five non-towered airports. Integrating deep-neural-network-based aircraft detectors and image-correlation-based aircraft trackers achieved an accuracy of about 95%, while ensuring processing times that are needed for real-time implementation.

Risky Driving Behavior Recognition Based on Vehicle Trajectory.

This paper proposes a measurement of risk (MOR) method to recognize risky driving behavior based on the trajectory data extracted from surveillance videos. Three types of risky driving behavior are studied in this paper, i.e., speed-unstable driving, serpentine driving, and risky car-following driving. The risky driving behavior recognition model contains an MOR-based risk evaluation model and an MOR threshold selection method. An MOR-based risk evaluation model is established for three types of risky driving behavior based on driving features to quantify collision risk. Then, we propose two methods, i.e., the distribution-based method and the boxplot-based method, to determine the threshold value of the MOR to recognize risky driving behavior. Finally, the trajectory data extracted from UAV videos are used to validate the proposed model. The impact of vehicle types is also taken into consideration in the model. The results show that there are significant differences between threshold values for cars and heavy trucks when performing speed-unstable driving and risky car-following driving. In addition, the difference between the proportion of recognized risky driving behavior in the testing dataset compared with that in the training dataset is limited to less than 3.5%. The recognition accuracy of risky driving behavior with the boxplot- and distribution-based methods are, respectively, 91% and 86%, indicating the validation of the proposed model. The proposed model can be widely applied to risky driving behavior recognition in video-based surveillance systems.

An Efficient Approach for Detecting and Classifying Moving Vehicles in a Video Based Monitoring System

Moving objects detection, type recognition, and traffic analysis in video-based surveillance systems is an active area of research which has many applications in road traffic monitoring. This paper is on using classical approaches of image processing to develop an efficient algorithm for computer vision based on traffic surveillance system that can detect and classify moving vehicles, besides serving some other traffic analysis issues like finding vehicles speed and heading, tracking specified vehicles, and finding traffic load. The algorithm is designed to be flexible for modification to fulfill the changes in design objectives, having limited computation time, giving good accuracy, and serves inexpensive implementation. A 92% of success is achieved for the considered test, with the missed cases being abnormal that are not defined to the algorithm. The computation time, with a platform (hardware and software) dependent, the algorithm took to produce results was parts of milliseconds. A CNN based deep learning classifier was built and evaluated to judge the feasibility of involving a modern approach in the design for the targeted aims in this work. The modern NN based deep learning approach is very powerful and represents the choice for many very sophisticated applications, but when the purpose is restricted to limited requirements, as it is believed the case is here, the reason will be to use the classical image processing procedures. In making choice, it is important to consider, among many things, accuracy, computation time, and simplicity of design, development, and implementation.

Online learning of contexts for detecting suspicious behaviors in surveillance videos

One of the main incentives for implementing video-based surveillance systems is the urban security. During the last years, several approaches for automatic detection of suspicious events have been proposed. Those methods usually require a training stage before starting their operation. This means that previous to run time a representative dataset of interest events, that may occur in the future, must be available. Nevertheless, most real surveillance systems lack of that information, so many of those proposals results impractical.In this paper, a context online learning scheme for detecting suspicious behaviors on surveillance videos is proposed. Contextual information, which is inferred from videos of people in a scenario, allows detecting suspicious behaviors before an eventual criminal's final attack occur. The main attribute of the proposed approach is the capacity to start up its operation with a reduced training dataset. By an incremental learning process, which uses new data obtained during the online operation, the proposed scheme improves the performance over time achieving a better adaptation to conditions of each scenario.The proposed scheme was validated on two datasets. The first of them includes threats against a parked truck and its driver. The second testing dataset is composed of night assault scenes recorded in an urban environment. The experimental results demonstrate that the proposed method is able to learn incrementally from a reduced initial dataset, achieving a performance similar to batch-type systems trained with all data simultaneously and outperforming five state-of-the-art algorithms over violence detection.

A simple low-cost video-based surveillance system for a flash flood warning system

Flash flood is the most common and frequent natural disasters worldwide and can cause a loss of life and damage to property and the environment. In 2014, Manado city has experienced the most devastating flood that caused 19 people died, 10,884 homes destroyed, 2,091 people became homeless and estimated an economic loss of US$ 124 million. The huge loss was caused by the people and government did not receive information about water level rising on the watershed of the Tondano River. In conventional, decision maker needs the precipitation data and critical threshold value to issue the evacuation. There is an urgent requirement for end-to-end solution where the flash flood early warning system (FEWS) which can processed data in real-time can be integrated with Video-Based Surveillance System to take a role in saving peoples life. The aim of this study was to create a stand-alone system (Sensor Node) that could be placed on the watershed to monitor the water level rising using a microcomputer, a camera and a distance sensor. The aim of the system proposed was to measure the water level rising and when the threshold value was reached, the system would capture and display a real-time video. Future research can be done to “go-live” and posted the captured real-time video on the YouTube or Facebook in which people used them widely as a social media

Locally Statistical Dual-Mode Background Subtraction Approach

Due to the variety of background model in the real world, detecting changes in a video cannot be addressed exhaustively by a simple background subtraction method, especially with several motion detection challenges, such as dynamic background, camera jitter, intermittent object motion, and so on. In this paper, we propose an efficient background subtraction method, namely locally statistical dual-mode (LSD), for detecting moving objects in video-based surveillance systems. The method includes a local intensity pattern comparison algorithm for foreground segmentation by analyzing the homogeneity of intensity patterns of the input frame and the background model, in which the homogeneity is calculated by the mean and standard deviation of pixel intensity. Besides that, a dual-mode scheme is developed to temporally update the background model for the short- and long-term scenarios corresponding to sudden and gradual changes in the background. The advantage of this scheme is the allowance of updating the model in both pixel- and frame-wise manners simultaneously. The parameters used in both the local intensity pattern comparison algorithm and the dual-mode background model updating scheme are estimated for every input frame consecutively based on local and global statistical information of segmentation result. In experiments, the proposed LSD method is extensively evaluated on the Wallflower and CDnet2014 datasets; and remarkable performance demonstrates its preeminence to the many state-of-the-art background subtraction approaches in terms of segmentation accuracy and computational complexity.

A novel infrared video surveillance system using deep learning based techniques

This paper presents a new, practical infrared video based surveillance system, consisting of a resolution-enhanced, automatic target detection/recognition (ATD/R) system that is widely applicable in civilian and military applications. To deal with the issue of small numbers of pixel on target in the developed ATD/R system, as are encountered in long range imagery, a super-resolution method is employed to increase target signature resolution and optimise the baseline quality of inputs for object recognition. To tackle the challenge of detecting extremely low-resolution targets, we train a sophisticated and powerful convolutional neural network (CNN) based faster-RCNN using long wave infrared imagery datasets that were prepared and marked in-house. The system was tested under different weather conditions, using two datasets featuring target types comprising pedestrians and 6 different types of ground vehicles. The developed ATD/R system can detect extremely low-resolution targets with superior performance by effectively addressing the low small number of pixels on target, encountered in long range applications. A comparison with traditional methods confirms this superiority both qualitatively and quantitatively.

Classification of Multidimensional Time-Evolving Data Using Histograms of Grassmannian Points

In this paper, we address the problem of classifying multidimensional time-evolving data in dynamic scenes. To take advantage of the correlation between the different channels of data, we introduce a generalized form of a stabilized higher order linear dynamical system (sh-LDS) and we represent the multidimensional signal as a third-order tensor. In addition, we show that the parameters of the proposed model lie on a Grassmann manifold and we attempt to address the classification problem through study of the geometric properties of the sh-LDS’s space. Moreover, to tackle the problem of nonlinearity of the observation data, we represent each multidimensional signal as a cloud of points on the Grassmann manifold and we create a codebook by identifying the most representative points. Finally, each multidimensional signal is classified by applying a bag-of-systems approach having first modeled the variation of the class of each codeword on its tangent space instead of the sh-LDS’s space. The proposed methodology is evaluated in three different application domains, namely, video-based surveillance systems, dynamic texture categorization, and human action recognition, showing its great potential.

Shadow-Free, Expeditious and Precise, Moving Object Separation from Video

The foreground–background separation is an essential part of any video-based surveillance system. Gaussian Mixture Models (GMM) based object segmentation method accurately segments the foreground, but it is computationally expensive. In contrast, single Gaussian-based segmentation is computationally inexpensive but inaccurate because it can not handle the variations in the background. There is a trade-off between computation efficiency and precision in the segmentation approach. From the experimental observations, the variations such as lighting variations, shadows, background motion, etc., affect only a few pixels in the frames in temporal direction. So, unaffected pixel can be modeled by single Gaussian in temporal direction while the affected pixels may need GMM to handle the variations in the background. We propose an adaptive algorithm which models pixel dynamically in terms of number of Gaussians in temporal direction. The proposed method is computationally inexpensive and precise. The flexibility in terms of number of Gaussians used to model each pixel, along with adaptive learning approach, reduces the time complexity of the algorithm significantly. To resolve spacial occlusion problem, a spatial smoothing is carried out by weighted [Formula: see text] nearest neighbors which improves the overall accuracy of proposed algorithm. To avoid false detection due to illumination variations and shadows in a particular image, illumination invariant representation is used.

Assessment of radiation dose received by nuclear plant personnel through a video-based surveillance system

This work aims at developing a video-based surveillance system for safety purposes in nuclear plants. The objective is to assess the radiation dose received by nuclear plant personnel, while they execute daily tasks, by means of computer vision methods. The system is conceived to provide some redundancy to the radioprotection means currently in use, being independent and complementary to them. After evaluating some methods from the literature for automatic target detection and tracking, a novel system is developed to correctly detect, track and identify people, so that the radiation dose received by each person is reliably computed. The video data are supplied by cameras installed in the nuclear plant room. Radiation dose rate mapping is combined with the tracking results to account for the received doses. We provide experimental results from a research reactor room, which show that the proposed system achieves radiation dose estimates that are in general similar to the ones of the ground truth. The database developed in this work for performance evaluation has been made publicly available for the research community.

Gait recognition based on Gabor wavelets and (2D)2PCA

Gait recognition is one of the most important techniques in application areas such as video-based surveillance, human tracking and medical systems. In this study, a novel Gabor wavelets based gait recognition algorithm is proposed, which consists of three steps. First, the gait energy image (GEI) is formed by extracting different orientation and scale information from the Gabor wavelet. Secondly, A two-dimensional principal component analysis ((2D)2PCA) method is employed to reduce the feature space dimension. The (2D)2PCA method minimizes the within-class distance and maximizes the between-class distance. Last, the multi-class support vector machine (SVM) is adopted to recognize different gaits. Experimental results performed on CASIA gait database show that the proposed gait recognition algorithm is generally robust, and provides higher recognition accuracy comparing with existing methods.

Hierarchical and Networked Vehicle Surveillance in ITS: A Survey

Traffic surveillance has become an important topic in intelligent transportation systems (ITSs), which is aimed at monitoring and managing traffic flow. With the progress in computer vision, video-based surveillance systems have made great advances on traffic surveillance in ITSs. However, the performance of most existing surveillance systems is susceptible to challenging complex traffic scenes (e.g., object occlusion, pose variation, and cluttered background). Moreover, existing related research is mainly on a single video sensor node, which is incapable of addressing the surveillance of traffic road networks. Accordingly, we present a review of the literature on the video-based vehicle surveillance systems in ITSs. We analyze the existing challenges in video-based surveillance systems for the vehicle and present a general architecture for video surveillance systems, i.e., the hierarchical and networked vehicle surveillance, to survey the different existing and potential techniques. Then, different methods are reviewed and discussed with respect to each module. Applications and future developments are discussed to provide future needs of ITS services.

Evaluation of the Benefit-to-Cost of Freeway Service Patrol in Advanced Video Surveillance System Environment: A Case Study in China

Abstract With the growing traffic incidents led by the rapid increase of traffic volume on freeways, Chinese government is facing how to ensure the efficiency and safety of the freeway system. One of the hot debates is the efficiency of Freeway service patrols (FSPs) technology in the advanced video surveillance system environment, namely dense video-based surveillance system (DVBSS). To help making a correct decision, this paper develops an effective approach to assess the effectiveness of FSPs in the DVBSS environment. This paper proposes a simulation method to measure the B/C ratios of FSPs in advanced VSS environment. B/C ratios are evaluated under four scenarios namely basic scenario, FSP1, FSP2, and FSP3. All scenarios were simulated in the 9.4-kilometer segment and data were collected from the working log of traffic monitoring center. Among all scenarios, B/C ratio of FSP1 is lowest, sometimes less than1, indicating that not all the FSPs scenarios are effective in the advanced VSS environment. However, the B/C ratios of FSP2 and FSP3 in mean cost scenarios exceed 1, represent as 4.33:1 and 3.47:1 respectively. It indicated that FSPs technology is still an effective way for traffic incident management in VSS environment. To explore the robustness of the results, authors extended the length of patrolling segment to 13.4 kilometers.

A novel framework for intelligent surveillance system based on abnormal human activity detection in academic environments

Abnormal activity detection plays a crucial role in surveillance applications, and a surveillance system that can perform robustly in an academic environment has become an urgent need. In this paper, we propose a novel framework for an automatic real-time video-based surveillance system which can simultaneously perform the tracking, semantic scene learning, and abnormality detection in an academic environment. To develop our system, we have divided the work into three phases: preprocessing phase, abnormal human activity detection phase, and content-based image retrieval phase. For motion object detection, we used the temporal-differencing algorithm and then located the motions region using the Gaussian function. Furthermore, the shape model based on OMEGA equation was used as a filter for the detected objects (i.e., human and non-human). For object activities analysis, we evaluated and analyzed the human activities of the detected objects. We classified the human activities into two groups: normal activities and abnormal activities based on the support vector machine. The machine then provides an automatic warning in case of abnormal human activities. It also embeds a method to retrieve the detected object from the database for object recognition and identification using content-based image retrieval. Finally, a software-based simulation using MATLAB was performed and the results of the conducted experiments showed an excellent surveillance system that can simultaneously perform the tracking, semantic scene learning, and abnormality detection in an academic environment with no human intervention.

Background modeling using Object-based Selective Updating and Correntropy adaptation

Background modeling is widely used in visual surveillance systems aiming to facilitate analysis of real-world video scenes. The goal is to discriminate between pixels from foreground objects and those ones from the background. However, real-world scenarios tend to have time and spatial non-stationary variations, being difficult to reveal the foreground and background entities from video data. Here, we propose a novel adaptive background modeling, termed Object-based Selective Updating with Correntropy (OSUC), to support video-based surveillance systems. Our approach that is developed within an adaptive learning framework unveils existing spatio-temporal pixel relationships, making use of a single Gaussian for the model representation stage. Moreover, we introduce a background updating scheme composed of an updating rule that is based on the stochastic gradient algorithm and Correntropy cost function. As a result, this scheme can extract the temporal statistical pixel distribution, at the same time, dealing with non-stationary pixel value fluctuations that affect the background model. Here, an automatic tuning strategy of the cost function bandwidth parameter is carried out that can handle both Gaussian and non-Gaussian noise environments. Besides, to include pixel spatial relationships in the background modeling processing, we introduce an object-based selective learning rate strategy for enhancing the background modeling accuracy. Particularly, an object motion analysis stage is presented to detect and track foreground entities based on pixel intensities and motion direction attained via optical flow computation. Testing is provided on well-known datasets for discriminating between foreground and background that include stationary and non-stationary behaviors. Achieved results show that the OSUC outperforms, in most of the considered cases, the-state-of-the-art approaches with an affordable computational cost. Therefore, the proposed approach is suitable for supporting real-world video-based surveillance systems.

A Distribution Model with Pattern Structure in Formal Concept Analysis for Meteorological Data Minging

Automatic gender classification of an individual can be very useful in video-based surveillance systems and human-computer interaction systems. Currently, gait from a single viewpoint has been used to recognize the gender of a person. Considering the multiple cameras used in real environments, we investigate gender classification from human gait by using multi-view fusion, a relatively understudied problem. In this paper, we present a new approach to integrate information from multi-view gait at the feature level. First, gait energy images (GEI) are constructed from the video streams for different viewpoints. Then, the feature fusion is performed by putting GEI images and camera views together to generate a third-order tensor (x, y, view). A multi-linear principal component analysis (MPCA) is employed to reduce dimensionality of the tensor objects which integrate all views. The proposed fusion scheme is tested on CASIA database and compared with other fusion methods. The experimental results show that MPCA based feature fusion is quite effective for multi-view gait based gender classification.

Generic Learning-Based Ensemble Framework for Small Sample Size Face Recognition in Multi-Camera Networks.

Multi-camera networks have gained great interest in video-based surveillance systems for security monitoring, access control, etc. Person re-identification is an essential and challenging task in multi-camera networks, which aims to determine if a given individual has already appeared over the camera network. Individual recognition often uses faces as a trial and requires a large number of samples during the training phrase. This is difficult to fulfill due to the limitation of the camera hardware system and the unconstrained image capturing conditions. Conventional face recognition algorithms often encounter the “small sample size” (SSS) problem arising from the small number of training samples compared to the high dimensionality of the sample space. To overcome this problem, interest in the combination of multiple base classifiers has sparked research efforts in ensemble methods. However, existing ensemble methods still open two questions: (1) how to define diverse base classifiers from the small data; (2) how to avoid the diversity/accuracy dilemma occurring during ensemble. To address these problems, this paper proposes a novel generic learning-based ensemble framework, which augments the small data by generating new samples based on a generic distribution and introduces a tailored 0–1 knapsack algorithm to alleviate the diversity/accuracy dilemma. More diverse base classifiers can be generated from the expanded face space, and more appropriate base classifiers are selected for ensemble. Extensive experimental results on four benchmarks demonstrate the higher ability of our system to cope with the SSS problem compared to the state-of-the-art system.

APPLYING EDGE DENSITY BASED REGION GROWING WITH FRAME DIFFERENCE FOR DETECTING MOVING OBJECTS IN VIDEO SURVEILLANCE SYSTEMS

Detecting moving object is the key step in Video based Surveillance Systems. This paper presents a unique approach for detecting moving object from a video sequence based on edge detection and Edge Density Based Region Growing. To start with the video sequence is sub-divided into its constituent frames. The method proceeds by considering two consecutive frames at time intervals (t-1) and t. Then after we use Sobel’s operator to detect the edges present in respective frames. Detecting edges helps in estimating the object boundaries. After this we perform frame difference which eliminates most of the edges belonging to background. To highlight the regions containing the moving object we apply Edge Density Based Region Growing algorithm on the edge difference image. At last using Connected Component Analysis and Morphological Opening we extract the exact moving object. Experimental results show that the method is robust enough to detect the moving objects with complex backgrounds.