Real-time Video Surveillance Research Articles

Moving Object Detection and Recognition Using Optical Flow and Eigen Face Using Low Resolution Video

Background: As crime is increasing day by day, various applications are proposed to protect public places. Monitoring and tracking of video surveillance system is the most difficult task and it is prominent that human beings are not reliable and efficacious in doing this job. Objective: The prime objective of this research is to develop an automatic monitoring and inspecting system that is competent enough to detect and track the moving objects in real-time using a low-resolution video surveillance camera. Methods: Firstly, the video data acquired from a low-resolution video surveillance camera is used for generating RGB video frames which are converted into gray scale. Optical flow and Eigen face algorithms are applied to extract and match the moving object in the video sequence with the images stored in the database. Results: The proposed system is compared with the already existing systems and it is observed that this approach gives more accurate results. This system can meet the requirement of real-time tracking even when the targeted image resolution is smaller than 160x120. Conclusion: This method uses optical flow and Eigen face algorithm to track and detect the moving objects. The system gives high performance and can be used for real time object tracking. The same experiment can be applied for the human faces too.

On the crack characterization of reinforced concrete structures: Experimental and data-driven numerical study

In today’s rapid infrastructure development, reinforced concrete (RC) construction accounts for majority of the total residential, commercial and industrial projects. These RC structures, once constructed, deteriorates significantly during their service life. The deterioration of RC structural elements increases mainly with the ageing and sustained loading on the structure and depends on various parameters. One of the primary parameter to access the serviceability of an RC structure is cracking. In this work, an experimental and a numerical study is carried out on the flexural crack behaviour of reinforced concrete beam members. The experimental investigation is focused on the effect of flexural crack by varying the percentage of tensile steel on beam sections. Later, a computer vision-based data-driven numerical tool for crack representation and quantification is developed and validated based on the real-time spatio-temporal video surveillance data of the flexural testing on beams. The developed numerical tool based on the experimental study can be used by the structural designers and practitioners for a robust serviceability-based design of reinforced concrete members and systems.

Design and PoC Implementation of Mmwave-Based Offloading-Enabled UAV Surveillance System

Among UAV applications, real-time high-resolution video surveillance systems using unmanned aerial vehicles (UAVs) especially gain our research interests, because high mobilities of UAVs and high qualities of videos can dramatically improve the surveillance performance and evolve the system, especially for artificial intelligence (AI)-based monitoring systems. However, it is challenging to transmit high-resolution videos to ground station in real-time and also impossible to perform AI detection based on these videos on UAVs. To address the challenges, we propose to take full advantages of mmWave communications and to apply it into UAV-based surveillance systems to transmit ultra-high-quality and ultra-high-resolution uncompressed 4K videos. To increase energy efficiency of UAVs, we propose to offload all computations via mmWave from UAVs to ground station. Moreover, in addition to design of system architecture, proof-of-concept (PoC) prototype hardware of the proposed system is practically developed to demonstrate technical feasibilities of mmWave communication on UAVs and the enhancements to practical surveillance applications. The practical issues in the implementation and the necessary hardware/software applied in the prototype system are explained in detail in this paper. An outdoor PoC experiment is conducted, and the experimental results are consistent with design, so that the effectiveness of the system is confirmed.

Detection and Location of Safety Protective Wear in Power Substation Operation Using Wear-Enhanced YOLOv3 Algorithm

Wearing personal safety protective equipment (PSPE) plays a key role in reducing electrical injuries to electrical workers. However, substation employees often ignore this regulation due to lack of safety awareness and discomfortable feeling of wearing PSPE. Therefore, it is necessary to develop a detection algorithm for PSPE and workers to build real-time video surveillance systems in power substations. In this paper, a wear-enhanced YOLOv3 method for real-time detection of PSPE and substation workers is proposed. The gamma correction is first applied as the preprocessing method to highlight the details of the operators and data augmentation is performed. Next, K-means++ algorithm replaces K-means in wear-enhanced YOLOv3 method to derive the most suitable prior bounding box size and lift the detection speed. Then, the proposed method can be quickly and effectively trained based on transfer learning. Finally, extensive experiments are carried out on a dataset of images about usage of safety helmets, insulating gloves and boots. Using the proposed method, the mean average precision is improved by over 2% and the frames per second is the highest compared with other typical object detection methods, which illustrates the effectiveness of the wear-enhanced YOLOv3 method for PSPE and workers detection.

A New System for Real-time Video Surveillance in Smart Cities Based on Wireless Visual Sensor Networks and Fog Computing

Nowadays, public security is becoming an increasingly serious issue in our society and its requirements have been extended from urban centers to all remote areas. Therefore, surveillance and security cameras are being deployed worldwide. Wireless Visual Sensor Networks nodes can be employed as camera nodes to monitor in the city without the need for any cables installation. However, these cameras are constrained in processing, memory, and energy resources. Also, they generate a massive amount of data that must be analyzed in real-time to ensure public safety and deal with emergency situations. As a result, data processing, information fusion, and decision making have to be executed on-site (near to the data collection location). Besides, surveillance cameras are directional sensors, which makes the coverage problem another issue to deal with. Therefore, we present a new system for real-time video surveillance in a smart city, in which transportations equipped with camera nodes are used as the mobile part of the system and an architecture based on fog computing and wireless visual sensor networks is adopted. Furthermore, we propose an approach for selecting the camera nodes that will participate in the tracking process and we simulated three different use cases to test the effectiveness of our system in terms of target detection. The simulation results show that our system is a promising solution for smart city surveillance applications.

Crowd density detection method based on crowd gathering mode and multi-column convolutional neural network

Crowds and stampedes often occur in crowd gathering places, resulting in a large number of casualties and causing great negative social impacts. Traditional research on the dynamic assessment of crowd gathering safety mainly relies on real-time video monitoring, but lacks reliable methods for processing a large amount of video data from different sources, different perspectives and different granularities. Based on Edward Hall's personal space theory, this article considers crowd psychology and other factors, and establishes static basic model of crowd gathering patterns. In order to fuse real-time multi-granularity surveillance videos with different perspectives, a multi-column convolutional neural network (M-CNN) was used to extract the local density characteristics of the crowd in a low-altitude perspective, thereby establishing a holographic model of the temporal and spatial evolution of the crowd situation, and a new crowd gathering safety assessment method. This method was actually applied to the safety assessment of crowd gathering in Suzhou landmark-Urban Living Fountain Square, and achieved good results, providing theoretical support for the safety management of crowd gathering places.

A fast and robust real-time surveillance video stitching method

Real-time video stitching can build a wider field of view for surveillance, which faces a compromise between stitching speed and visual quality. A fast and robust real-time surveillance video stitching method is proposed to deal with the ghosting effect caused by moving objects and misalignments caused by background change or slight camera shift through automatic updating. By stitching key frames, parameters such as pix mapping table, stitching seams and blending weights are calculated, and most of subsequent frames are directly blended with CUDA acceleration based on the pre-calculated stitching parameters. Fast and effective algorithms are designed to detect the change of stitching seam and background during the whole stitching process, which determines whether to update the stitching seam or recalculate stitching parameters. Experiments show that this method can robustly and automatically solve the ghosting and misalignments to improve visual quality and achieve satisfactory real-time performance.

Real time object detection and trackingsystem for video surveillance system

This paper introduces a system capable of real-time video surveillance in low-end edge computing environment by combining object detection tracking algorithm. Recently, the accuracy of object detection has been improved due to the performance of approaches based on deep learning algorithm such as region-based convolutional network, which has two stages for inferencing. One-stage detection algorithms such as single shot detector and you only look once (YOLO) have been developed at the expense of some accuracy and can be used for real-time systems. However, high-performance hardware such as general-purpose graphics processing unit is required to achieve excellent object detection performance and speed. In this study, we propose an approach called N-YOLO which is instead of resizing image step in YOLO algorithm, it divides into fixed size images used in YOLO and merges detection results of each divided sub-image with inference results at different times using correlation-based tracking algorithm the amount of computation for object detection and tracking can be significantly reduced. In addition, we propose a system that can guarantee real-time performance in various edge computing environments by adaptively controlling the cycle of object detection and tracking.

Analytical Modelling of Video Surveillance System Along Forest Railway Tracks

An analytical model has been developed by the authors for determining the angular alignment, position, length related parameters of a proposed real-time video surveillance system to be installed along railway tracks or highways passing through open or moderately dense or very dense forest regions. The proposed system is capable of providing 24 × 7 real-time surveillance of the railway tracks or highways and its surroundings in order to avoid accidents caused by animal-train or animal-vehicle collisions.

Implementation and analysis of MultiCode MultiCarrier Code Division Multiple Access (MC–MC CDMA) in IEEE 802.11ah for UAV Swarm communication

Utilization of flying platforms like Unmanned Aerial Vehicles (UAVs) is progressively increasing in our daily lives. Key characteristics such as flexibility, mobility and adaptive altitude, make UAVs particularly suitable for application in areas such as crop management, goods delivery, public safety and real-time video surveillance of public places to assist in terrestrial networks. IEEE 802.11ah is a WiFi brand Internet of Things (IoT) technology, with properties of suitable data rates, flexibility in channelization, Multi Input Multi Output (MIMO) capability and low power consumption etc. Compared to the existing wireless technologies, IEEE 802.11ah is a suitable choice in terms of data rate and performance. In this research, MultiCode (MC) MultiCarrier (MC) Code Division Multiple Access (CDMA) is incorporated in the Physical (PHY) layer of IEEE 802.11ah to meet the differential requirements of range and throughput of UAV to UAV and UAV to Ground Control Station (GCS) communication links for UAV swarm. Depending upon the nature of data transmitted between the UAVs and GCS, suitable Modulation and Coding Schemes (MCS) are selected for multiple bandwidths. Performance analysis of MC–MC CDMA in IEEE 802.11ah has been performed in Additive White Gaussian Noise (AWGN). It has been observed that an increase/decrease in the modulation order directly affects data rate, with a compromised performance at higher modulation order.

Real-time smart video surveillance to manage safety: A case study of a transport mega-project

There is a tendency for accidents and even fatalities to arise when people enter hazardous work areas during the construction of projects in urban areas. A limited amount of research has been devoted to developing vision-based proximity warning systems that can determine when people enter a hazardous area automatically. Such systems, however, are unable to identify specific hazards and the status of a piece of plant (e.g., excavator) in real-time. In this paper, we address this limitation and develop a real-time smart video surveillance system that can detect people and the status of plant (i.e. moving or stationary) in a hazardous area. The application of this approach is demonstrated during the construction of a mega-project, the Wuhan Rail Transit System in China. We reveal that our combination of computer vision and deep learning can accurately recognize people in a hazardous work area in real-time during the construction of transport projects. Our developed systems can provide instant feedback concerning unsafe behavior and thus enable appropriate actions to be put in place to prevent their re-occurrence.

SAT: Single-Shot Adversarial Tracker

Deep learning-based tracking methods have shown favorable performance on multiple benchmarks. However, most of these methods are not designed for real-time video surveillance systems due to the complex online optimization process. In this article, we propose a single-shot adversarial tracker (SAT) to efficiently locate objects of interest in surveillance videos. Specifically, we propose a lightweight convolutional neural network-based generator, which fuses multilayer feature maps to accurately generate the target probability map (TPM) for tracking. To more effectively train the generator, an adversarial learning framework is presented. During the online tracking stage, the learned TPM generator can be directly employed to generate the target probability map corresponding to the searching region in a single shot. The proposed SAT can lead to the average tracking speed of 212 FPS on a single GPU, while still achieving the favorable performance on several popular benchmarks. Furthermore, we also present a variant of SAT by considering both scale estimation and online updating in SAT, which achieves better accuracy than SAT while still maintaining very fast tracking speed (i.e., exceeding 100 FPS).

Unleashing the Power of Participatory IoT with Blockchains for Increased Safety and Situation Awareness of Smart Cities

IoT emerges as an unprecedented paradigm with great potential for changing how people interact, think and live. It is making existing Internet services feasible in ways that were previously impossible, as well as paving the way for new situation-awareness applications suitable for smart cities, such as realtime video surveillance, traffic control, and emergency management. These applications will typically rely on large numbers of IoT devices to collect and collaboratively process streamed data to enable real-time decision making. In this paper, we introduce the concept of Semantic Virtual Space (SVS), an abstraction for virtualized cloud-enabled IoT infrastructure that is commensurate with the goals and needs of these emerging smart city applications, and propose and discuss scalable architectures and mechanisms that enable and automate the deployment and management of multiple SVS instances on top of the cloud-enabled IoT infrastructure.

A Novel Low Processing Time System for Criminal Activities Detection Applied to Command and Control Citizen Security Centers

This paper shows a Novel Low Processing Time System focused on criminal activities detection based on real-time video analysis applied to Command and Control Citizen Security Centers. This system was applied to the detection and classification of criminal events in a real-time video surveillance subsystem in the Command and Control Citizen Security Center of the Colombian National Police. It was developed using a novel application of Deep Learning, specifically a Faster Region-Based Convolutional Network (R-CNN) for the detection of criminal activities treated as “objects” to be detected in real-time video. In order to maximize the system efficiency and reduce the processing time of each video frame, the pretrained CNN (Convolutional Neural Network) model AlexNet was used and the fine training was carried out with a dataset built for this project, formed by objects commonly used in criminal activities such as short firearms and bladed weapons. In addition, the system was trained for street theft detection. The system can generate alarms when detecting street theft, short firearms and bladed weapons, improving situational awareness and facilitating strategic decision making in the Command and Control Citizen Security Center of the Colombian National Police.

Single Image Super-Resolution Algorithm Possessing Edge and Contrast Preservation

In real time surveillance video applications, it is often required to identify a region of interest in a degraded low resolution (LR) image. State-of-the-art super-resolution (SR) techniques produce images with poor illumination and degraded high frequency details. In this paper, we present a different approach for SISR by correcting the dual-tree complex wavelet transform (DT-CWT) subbands using the multi-stage cascaded joint bilateral filter (MSCJBF) and singular value decomposition (SVD). The proposed method exploits geometric regularity for implementing the covariance-based interpolation in the spatial domain. We decompose the interpolated LR image into different image and wavelet coefficients by employing DT-CWT. To preserve edges, we alter the wavelet sub-bands with the high frequency details obtained from the MSCJBF. Simultaneously, we retain uniform illumination by improving the image coefficients using SVD. In addition, the wavelet sub-bands undergo lanczos interpolation prior to the subband refinement. Experimental results demonstrate the effectiveness of our method.

Spatiotemporal Anomaly Detection Using Deep Learning for Real-Time Video Surveillance

Rapid developments in urbanization and autonomous industrial environments have augmented and expedited the need for intelligent real-time video surveillance. Recent developments in artificial intelligence for anomaly detection in video surveillance only address some of the challenges, largely overlooking the evolving nature of anomalous behaviors over time. Tightly coupled dependence on a known normality training dataset and sparse evaluation based on reconstruction error are further limitations. In this article, we propose the incremental spatiotemporal learner (ISTL) to address challenges and limitations of anomaly detection and localization for real-time video surveillance. ISTL is an unsupervised deep-learning approach that utilizes active learning with fuzzy aggregation, to continuously update and distinguish between new anomalies and normality that evolve over time. ISTL is demonstrated and evaluated on accuracy, robustness, computational overhead as well as contextual indicators, using three benchmark datasets. Results of these experiments validate our contribution and confirm its suitability for real-time video surveillance.

SCRAS Server-Based Crosslayer Rate-Adaptive Video Streaming over 4G-LTE for UAV-Based Surveillance Applications

This research focuses on intelligent unmanned aerial vehicle (UAV)-based, real-time video surveillance to ensure better monitoring and security of remote locations over 4G-LTE cellular networks by maximizing end-user quality of experience (QoE). We propose a novel server-based crosslayer rate-adaptive scheme (SCRAS) for real-time video surveillance over 4G-LTE networks using UAVs. Our key contributions are: (1) In SCRAS, mobile UAVs having preprogrammed flight co-ordinates act as servers, streaming real-time video towards a remote client; (2) server-side video rate adaptation occurs in 4G-LTE based on the physical characteristics of the received signal conditions due to variations in the wireless channel and handovers; (3) SCRAS is fully automated and independent of client assistance for rate adaptation, as it is intended for real-time, mission-critical surveillance applications; (4) SCRAS ensures that during rate adaptation, the current video frame should not be damaged by completing the current group of packet (GoP) before adaptation. Our simulations in NS3 provide credible evidence that SCRAS outperforms recently proposed schemes in providing better QoE for real-time, rate-adaptive, video surveillance over 4G-LTE under varying channel quality and frequent handovers that occur during flight by UAVs.

Analytics in real time surveillance video using two-bit transform accelerative regressive frame check

Face recognition is an established area of research in computer vision and it had played a great role in developing content based personal retrieval systems from real time surveillance video feeds. Face recognition in live videos is a complex problem as facial features fall into high dimensional space and involves large search time. Though, there is an extensive improvement in computational infrastructure over the years, the need for improved search algorithms without increase in cost is a challenge. Existingmethodologies in literature fail to perform in real time scenarios as the cost of feature matching is high. Hence, this research work proposes a Two-Bit Transform AccelerativeRegressive Frame Check algorithm (2BT-ARFCA) methodology that facilitates face recognition in video at a faster rate, suitable for surveillance and authentication applications. Finally the results are experimentally validated with variousvideo datasets and the state-of-the-art techniques proves that the proposed method performs better in terms of Specificity, Sensitivity, Mean Square Error (MSE), Peak signal to noise Ratio (PSNR), The Structural Similarity Index (SSIM) and accuracy.

Smart Real-Time Video Surveillance Platform for Drowsiness Detection Based on Eyelid Closure

We propose drowsiness detection in real-time surveillance videos by determining if a person’s eyes are open or closed. As a first step, the face of the subject is detected in the image. In the detected face, the eyes are localized and filtered with an extended Sobel operator to detect the curvature of the eyelids. Once the curves are detected, concavity is used to tell whether the eyelids are closed or open. Consequently, a concave upward curve means the eyelid is closed whereas a concave downwards curve means the eye is open. The proposed method is also implemented on hardware in order to be used in real-time scenarios, such as driver drowsiness detection. The evaluation of the proposed method used three image datasets, where images in the first dataset have a uniform background. The proposed method achieved classification accuracy of up to 95% on this dataset. Another benchmark dataset used has significant variations based on face deformations. With this dataset, our method achieved classification accuracy of 70%. A real-time video dataset of people driving the car was also used, where the proposed method achieved 95% accuracy, thus showing its feasibility for use in real-time scenarios.

Embedded Real-Time Video Surveillance System based on Multi-Sensor and Visual Tracking

Embedded Real-Time Video Surveillance System based on Multi-Sensor and Visual Tracking