Real Surveillance Research Articles

A Hybrid CNN-LSTM Model for Aircraft 4D Trajectory Prediction

The 4D trajectory is a multi-dimensional time series with plentiful spatial-temporal features and has a high degree of complexity and uncertainty. Aiming at these features of aircraft flight trajectory and the problem that it is difficult for existing trajectory prediction methods to extract spatial-temporal features from the trajectory data at the same time, we propose a novel 4D trajectory prediction hybrid architecture based on deep learning, which combined Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). An 1D convolution is used to extract the spatial dimension feature of the trajectory, and LSTM is used to mine the temporal dimension feature of the trajectory. Hence the high-precision prediction of the 4D trajectory is realized based on the sufficient fusion of the above features. We use real Automatic Dependent Surveillance -Broadcast (ADS-B) historical trajectory data for experiments and compare the proposed method with a single LSTM model and BP model on the same data set. The experimental results show that the trajectory prediction accuracy of the CNN-LSTM hybrid model is superior to a single model. The prediction error is reduced by an average of 21.62% compared to the LSTM model and by an average of 52.45% compared to the BP model. It provides a certain reference for the trajectory prediction research and Air Traffic Management decision-making.

Skeleton Based Abnormal Behavior Recognition Using Spatio-Temporal Convolution and Attention-Based LSTM

Abstract Behavior recognition is a well-known computer version technology, it has been used in many applications such as video surveillance, robotics, human-computer interaction and sports video, etc. However, most of existing works ignored the depth information so that they resulted in over-fitting and the inferior performance. Consequently, a novel framework for behavior recognition is proposed which is based on spatio-temporal convolution and attention-based LSTM (ST-CNN & ATT-LSTM). In this framework, deep spatial information is merged into each segment, meanwhile we focus on the method of key information extraction, which is essential for improving behavior recognition performance. Finally, the proposed framework is evaluated with real world surveillance video data, and the results indicate that our framework is superior to existing methods.

Real-time surveillance of infectious diseases and other health conditions during the arbaeenia mass gathering, Kerbala, Iraq, 2016: a teaching case-study

The Arbaeenia mass gathering (MG) is considered the largest religious MG in Iraq and occurs annually in the city of Kerbala with participation of several million visitors from different countries and most Iraqi provinces. This MG places attendants at a significant risk for contracting communicable diseases, sustaining injuries, and facing complications of health problems related to chronic diseases due to overcrowding and intensive physical activity. The goal of this case study to build the capacity of trainees on how to improve the surveillance systems for infectious diseases and other health conditions during MGs. This case study is based on the implementation of a real time surveillance system for illness and health problems affecting people attending the Arbaeenia MG in 2016 which enabled the surveillance team in Iraq to rapidly detect and respond to disease outbreaks. This case study teaches students how MGs offer an opportunity to improve existing surveillance systems or establish new ones. The case study is designed for training Novice field epidemiology trainees. The case study can be administered in 3-4 hours. Used as adjunct training material, the case study provides the trainees with competencies to use real time technology for surveillance data collection, analysis and timely response during the event.

Real-time decoding of automatic dependent surveillance – Broadcast signals by using software-defined radio and MATLAB®

In telecommunications engineering degrees, simulations are used to understand better theoretical concepts. However, in courses related to digital communications or signal processing usually these simulations cannot be easily applied to real data or in real scenarios. This limitation does not allow the students to verify that the learned theoretical hypotheses are valid in practice. Nowadays, software defined radio platforms have been proposed to validate theoretical telecommunications concepts in real situations. Therefore, by combining a software defined radio and simulation tools, students can implement the learned notions over real data in real scenarios. This allows students to work under real applications in order to learn by experience and to see that the theoretical concepts are valid in the real world. This paper presents a project involving real-time decoding of automatic dependent surveillance–broadcast (ADS-B) signals, implemented by using a universal software radio peripheral software defined radio and MATLAB®. The universal software radio peripheral allows to receive real automatic dependent surveillance–broadcast signals sent by the aircraft. Then, by using MATLAB® tools, the automatic dependent surveillance–broadcast baseband signals are decoded. The students are guided through a manual to demodulate the real automatic dependent surveillance–broadcast signals and display in a map the aircraft trajectories. The evaluation is based on a competition between the students, considering execution time of their algorithms, and a final report of the project. This project allows to obtain a game-oriented and novel approach for teaching digital communications and signal processing. The implementation of this project in telecommunications engineering courses motivate the students, allowing them to understand better difficult theoretical concepts, and improve their computer, teamwork, and collaboration skills. The originality of the approach is based on the usage of a prototyping platform, combined with simulation tools, for practical application of telecommunications engineering concepts using real signals.

Single Image Super-Resolution Algorithm Possessing Edge and Contrast Preservation

In real time surveillance video applications, it is often required to identify a region of interest in a degraded low resolution (LR) image. State-of-the-art super-resolution (SR) techniques produce images with poor illumination and degraded high frequency details. In this paper, we present a different approach for SISR by correcting the dual-tree complex wavelet transform (DT-CWT) subbands using the multi-stage cascaded joint bilateral filter (MSCJBF) and singular value decomposition (SVD). The proposed method exploits geometric regularity for implementing the covariance-based interpolation in the spatial domain. We decompose the interpolated LR image into different image and wavelet coefficients by employing DT-CWT. To preserve edges, we alter the wavelet sub-bands with the high frequency details obtained from the MSCJBF. Simultaneously, we retain uniform illumination by improving the image coefficients using SVD. In addition, the wavelet sub-bands undergo lanczos interpolation prior to the subband refinement. Experimental results demonstrate the effectiveness of our method.

RFR-DLVT: a hybrid method for real-time face recognition using deep learning and visual tracking

ABSTRACT In this paper, we proposed a hybrid method based on Deep Learning (DL) and visual tracking, RFR-DLVT, to achieve effective face recognition (FR). First, video sequences are divided into reference frames (RFs) and non-reference frames (NRFs). Then, the target face is identified through the DL-based FR method in RFs. In the meantime, the Kernelized-correlation-filters-based visual tracking method is used in NRFs to speed up FR. Our proposed method is tested on common data sets and achieves better performance. Particularly, RFR-DLVT has an accuracy of 99.6% and an efficiency of 30 (FPS) in the real-time FR on real-life surveillance videos.

Face search in CCTV surveillance

BackgroundWe present a series of experiments on visual search in a highly complex environment, security closed-circuit television (CCTV). Using real surveillance footage from a large city transport hub, we ask viewers to search for target individuals. Search targets are presented in a number of ways, using naturally occurring images including their passports and photo ID, social media and custody images/videos. Our aim is to establish general principles for search efficiency within this realistic context.ResultsAcross four studies we find that providing multiple photos of the search target consistently improves performance. Three different photos of the target, taken at different times, give substantial performance improvements by comparison to a single target. By contrast, providing targets in moving videos or with biographical context does not lead to improvements in search accuracy.ConclusionsWe discuss the multiple-image advantage in relation to a growing understanding of the importance of within-person variability in face recognition.

Multi-View Vehicle Type Recognition With Feedback-Enhancement Multi-Branch CNNs

Vehicle type recognition (VTR) is a quite common requirement and one of the key challenges in real surveillance scenarios, such as intelligent traffic and unmanned driving. Usually coarse-grained and fine-grained VTRs are applied in different applications, and the challenge from multiple viewpoints is critical for both cases. In this paper, we propose a feedback-enhancement multi-branch CNN (FM-CNN) to solve the challenge in these two cases. The proposed FM-CNN takes three derivatives of an image as input and leverages the advantages of hierarchical details, feedback enhancement, model average, and stronger robustness to translation and mirroring. A single global cross-entropy loss is insufficient to train such a complex CNN and so we add extra branch losses to enhance feedbacks to each branch. Though reusing pre-trained parameters, we propose a novel parameter update method to adapt FM-CNN to task-specific local visual patterns and global information in new datasets. To test the effectiveness of FM-CNN, we create our own multi-view VTR (MVVTR) data set since there are no such data sets available. And, for fine-grained VTR, we use the CompCars data set. Compared with state-of-the-art classification solutions without special preprocessing, the proposed FM-CNN demonstrates better performance in both coarse-grained and fine-grained scenarios. For coarse-grained VTR, it achieves 94.9% Top-1 accuracy on the MVVTR data set. For fine-grained VTR, it achieves 91.0% Top-1 and 97.8% Top-5 accuracies on the CompCars data set.

Learning behaviour recognition based on multi-object image in single viewpoint

Though greatly significant for education evaluation and improvement the analysis of learning behaviour of students is, recognition of behaviour of students in the classroom surveillance based on one single image remains a challenging work due to problems including heavy occlusion, low resolution, small target size, large variability of camera viewpoints and significant perspective effect. In this paper, an innovative multiple-model method for recognizing students’ learning behaviour and counting the number of students in classroom scenes via detecting individuals’ head, which divides students’ learning behaviour into three categories according to the head posture, is proposed. Through viewing heads at different postures as targets of different categories, behaviour recognition problem is transformed into multi-target detection problem. As the density of students in different images varies dramatically, a multiple model consisting of three detection networks with different sized receptive fields and one switch net is applied that each detection network selects a large number of sub-windows with small stride size from input images to optimize the performance of the model in classroom scenes while the switch net is trained to predict the density of students in the input image and relays the classroom image to the corresponding detection network according to the students’ density prediction. This method is obtained to outperform other methods in terms of accuracy and speed in comparison with state-of-the-art methods on a real classroom surveillance dataset.

Incremental-Learning-Based Unsupervised Anomaly Detection Algorithm for Terminal Airspace Operations

The terminal airspace system is one of the most complex human-made dynamical systems with a continuously evolving structure. Modern aviation technologies in the terminal airspace record detailed flight and airport information, including the states of aircraft and the operating conditions of airports as time series datasets, through on-board and ground-based systems. Detecting anomalies in such datasets is emerging as a key problem to understand air transportation system complexity and behavior. This paper proposes an incremental-learning-based anomaly detection algorithm that generates anomaly detection models from surveillance data. The algorithm is developed in a recursive fashion, such that it uses the overnight recorded data to improve the temporal-logic-based anomaly detection models daily—thus it is termed as an overnight update algorithm. The proposed algorithm is demonstrated with real air traffic surveillance data comprising of arrival flights to LaGuardia, John F. Kennedy, and Newark airports, thereby detecting anomalies for arrivals in the terminal airspace of the New York metroplex. This is followed by some analysis and insights from the results of these extensive tests.

Cross-resolution face recognition with pose variations via multilayer locality-constrained structural orthogonal procrustes regression

In real video surveillance scenes, the extracted face regions generally have low-resolution (LR) and are sensitive to pose and illumination variations; these flaws undoubtedly degrade the subsequent recognition task. To overcome these challenges, we propose an approach named multilayer locality-constrained structural orthogonal Procrustes regression (MLCSOPR). The proposed MLCSOPR not only learns the pose-robust discriminative representation features to reduce the resolution gap between the LR image space and the high-resolution (HR) one but also strengthens the consistency between the LR and HR image space. In particular, several contributions are made in this paper: (i) Inspired by the orthogonal Procrustes problem (OPP), a matrix approximation is exploited to find an optimal correction between two data matrices. (ii) The nuclear norm constraint is applied to the reconstruction error to maintain the structural property. (iii) Based on the abovementioned learned resolution-robust representation features, a linear regression-based classification strategy is adopted to recognize the LR input face images. Experiments on commonly used face databases have shown the effectiveness of the proposed method on cross-resolution face matching with pose variations.

Optimal face templates: the next step in surveillance face recognition

The paper deals with surveillance face recognition in security applications such as surveillance camera systems or access control systems. Presented research is focused on enhancing recognition performance, reducing classification time and memory requirements. We aim to make it feasible to implement face recognition in end devices such as cameras, identification terminals or popular IoT devices. Therefore, we utilize algorithms that require low computational power and optimize them in order to reach higher recognition rates. We present a novel higher quantile method that enhances recognition performance via creation of robust and representative face templates for nearest neighbor classifier. Templates computed by the higher quantile method are determined by tolerance intervals which handle feature variability caused by face pose, expression, illumination and possible low image quality. The recognition performance evaluation has been conducted on images captured by surveillance camera system that are contained in unique IFaViD dataset. The IFaViD is the only one dataset captured by real surveillance camera system containing complex scenarios. The results show that the higher quantile method outperforms the contemporary approaches by 4%, respectively, 10% depending on the IFaViD’s test subset.

Online learning of contexts for detecting suspicious behaviors in surveillance videos

One of the main incentives for implementing video-based surveillance systems is the urban security. During the last years, several approaches for automatic detection of suspicious events have been proposed. Those methods usually require a training stage before starting their operation. This means that previous to run time a representative dataset of interest events, that may occur in the future, must be available. Nevertheless, most real surveillance systems lack of that information, so many of those proposals results impractical.In this paper, a context online learning scheme for detecting suspicious behaviors on surveillance videos is proposed. Contextual information, which is inferred from videos of people in a scenario, allows detecting suspicious behaviors before an eventual criminal's final attack occur. The main attribute of the proposed approach is the capacity to start up its operation with a reduced training dataset. By an incremental learning process, which uses new data obtained during the online operation, the proposed scheme improves the performance over time achieving a better adaptation to conditions of each scenario.The proposed scheme was validated on two datasets. The first of them includes threats against a parked truck and its driver. The second testing dataset is composed of night assault scenes recorded in an urban environment. The experimental results demonstrate that the proposed method is able to learn incrementally from a reduced initial dataset, achieving a performance similar to batch-type systems trained with all data simultaneously and outperforming five state-of-the-art algorithms over violence detection.

On mixed PARMA modeling of epidemiological time series data

A rich class of traditional statistical models and methods is currently available for the early detection of epidemic activity in epidemiological surveillance systems. Real time surveillance though, is often difficult to be fully achieved because of the seasonality involved in the series. Indeed, whenever the correlation structure of a series depends on the season, the time series involved fails to reach stationarity with all the associated modeling consequences. In such situations, a useful class of models is that of periodic auto-regressive moving average (PARMA) models allowing parameters that depend on season. In this work, for the modeling of influenza-like syndrome morbidity, the general form as well as special cases of PARMA models are considered, and via model selection identification and likelihood-based techniques, the optimal model is selected. Climatological and meteorological covariates associated with influenza-like syndrome are also incorporated into the model structure. The derived results are satisfactory since the selected model succeeds in identifying the epidemic waves, and in estimating accurately the influenza-like syndrome morbidity burden in the case of Greece (for the period 2014–2016).

A two-stage deep neural network for multi-norm license plate detection and recognition

In this work, we tackle the problem of multi-norm and multilingual license plate (LP) detection and recognition in natural scene images. The system architecture use a pipeline with two deep learning stages. The first network was trained to detect license plates on the full raw image by using the latest state-of-the-art deep learning based detector namely YOLOv2. The second stage is then applied on the cropped image to recognize captured license plate photographs. Two recognition engines are compared in this work: a segmentation-free approach based on a convolutional recurrent neural network where the recognition is carried out over the entire LP image without any prior segmentation and a joint detection/recognition approach that performs the recognition on the plate component level. We also introduced a new large-scale dataset for automatic LP recognition that includes 9.175 fully annotated images. In order to reduce the time and cost of annotation processing, we propose a new semi-automatic annotation procedure of LP images with labeled components bounding box. The proposed system is evaluated using two datasets collected from real road surveillance and parking access control environments. We show that the system using two YOLO stages performs better in the context of multi-norm and multilingual license plate. Additional experiments are conducted on the public AOLP dataset and show that the proposed approach outperforms over other existing state-of-the-art methods.

Dashboards as strategy to integrate multiple data streams for real time surveillance

Dashboards as strategy to integrate multiple data streams for real time surveillance

Streamlining Disease Surveillance System implementation in Tanzania: Lessons Learnt

Streamlining Disease Surveillance System implementation in Tanzania: Lessons Learnt

An Adaptive Framework for Multi-Vehicle Ground Speed Estimation in Airborne Videos

With the rapid development of unmanned aerial vehicles (UAVs), UAV-based intelligent airborne surveillance systems represented by real-time ground vehicle speed estimation have attracted wide attention from researchers. However, there are still many challenges in extracting speed information from UAV videos, including the dynamic moving background, small target size, complicated environment, and diverse scenes. In this paper, we propose a novel adaptive framework for multi-vehicle ground speed estimation in airborne videos. Firstly, we build a traffic dataset based on UAV. Then, we use the deep learning detection algorithm to detect the vehicle in the UAV field of view and obtain the trajectory in the image through the tracking-by-detection algorithm. Thereafter, we present a motion compensation method based on homography. This method obtains matching feature points by an optical flow method and eliminates the influence of the detected target to accurately calculate the homography matrix to determine the real motion trajectory in the current frame. Finally, vehicle speed is estimated based on the mapping relationship between the pixel distance and the actual distance. The method regards the actual size of the car as prior information and adaptively recovers the pixel scale by estimating the vehicle size in the image; it then calculates the vehicle speed. In order to evaluate the performance of the proposed system, we carry out a large number of experiments on the AirSim Simulation platform as well as real UAV aerial surveillance experiments. Through quantitative and qualitative analysis of the simulation results and real experiments, we verify that the proposed system has a unique ability to detect, track, and estimate the speed of ground vehicles simultaneously even with a single downward-looking camera. Additionally, the system can obtain effective and accurate speed estimation results, even in various complex scenes.

Object-Based Approach for Adaptive Source Coding of Surveillance Video

Intelligent analysis of surveillance videos over networks requires high recognition accuracy by analyzing good-quality videos that however introduce significant bandwidth requirement. Degraded video quality because of high object dynamics under wireless video transmission induces more critical issues to the success of smart video surveillance. In this paper, an object-based source coding method is proposed to preserve constant quality of video streaming over wireless networks. The inverse relationship between video quality and object dynamics (i.e., decreasing video quality due to the occurrence of large and fast-moving objects) is characterized statistically as a linear model. A regression algorithm that uses robust M-estimator statistics is proposed to construct the linear model with respect to different bitrates. The linear model is applied to predict the bitrate increment required to enhance video quality. A simulated wireless environment is set up to verify the proposed method under different wireless situations. Experiments with real surveillance videos of a variety of object dynamics are conducted to evaluate the performance of the method. Experimental results demonstrate significant improvement of streaming videos relative to both visual and quantitative aspects.

Analytics in real time surveillance video using two-bit transform accelerative regressive frame check

Face recognition is an established area of research in computer vision and it had played a great role in developing content based personal retrieval systems from real time surveillance video feeds. Face recognition in live videos is a complex problem as facial features fall into high dimensional space and involves large search time. Though, there is an extensive improvement in computational infrastructure over the years, the need for improved search algorithms without increase in cost is a challenge. Existingmethodologies in literature fail to perform in real time scenarios as the cost of feature matching is high. Hence, this research work proposes a Two-Bit Transform AccelerativeRegressive Frame Check algorithm (2BT-ARFCA) methodology that facilitates face recognition in video at a faster rate, suitable for surveillance and authentication applications. Finally the results are experimentally validated with variousvideo datasets and the state-of-the-art techniques proves that the proposed method performs better in terms of Specificity, Sensitivity, Mean Square Error (MSE), Peak signal to noise Ratio (PSNR), The Structural Similarity Index (SSIM) and accuracy.