Real-time Video Surveillance System Research Articles

Object detection and activity recognition in video surveillance using neural networks

PurposeThis paper aims to implement and extend the You Only Live Once (YOLO) algorithm for detection of objects and activities. The advantage of YOLO is that it only runs a neural network once to detect the objects in an image, which is why it is powerful and fast. Cameras are found at many different crossroads and locations, but video processing of the feed through an object detection algorithm allows determining and tracking what is captured. Video Surveillance has many applications such as Car Tracking and tracking of people related to crime prevention. This paper provides exhaustive comparison between the existing methods and proposed method. Proposed method is found to have highest object detection accuracy.Design/methodology/approachThe goal of this research is to develop a deep learning framework to automate the task of analyzing video footage through object detection in images. This framework processes video feed or image frames from CCTV, webcam or a DroidCam, which allows the camera in a mobile phone to be used as a webcam for a laptop. The object detection algorithm, with its model trained on a large data set of images, is able to load in each image given as an input, process the image and determine the categories of the matching objects that it finds. As a proof of concept, this research demonstrates the algorithm on images of several different objects. This research implements and extends the YOLO algorithm for detection of objects and activities. The advantage of YOLO is that it only runs a neural network once to detect the objects in an image, which is why it is powerful and fast. Cameras are found at many different crossroads and locations, but video processing of the feed through an object detection algorithm allows determining and tracking what is captured. For video surveillance of traffic cameras, this has many applications, such as car tracking and person tracking for crime prevention. In this research, the implemented algorithm with the proposed methodology is compared against several different prior existing methods in literature. The proposed method was found to have the highest object detection accuracy for object detection and activity recognition, better than other existing methods.FindingsThe results indicate that the proposed deep learning–based model can be implemented in real-time for object detection and activity recognition. The added features of car crash detection, fall detection and social distancing detection can be used to implement a real-time video surveillance system that can help save lives and protect people. Such a real-time video surveillance system could be installed at street and traffic cameras and in CCTV systems. When this system would detect a car crash or a fatal human or pedestrian fall with injury, it can be programmed to send automatic messages to the nearest local police, emergency and fire stations. When this system would detect a social distancing violation, it can be programmed to inform the local authorities or sound an alarm with a warning message to alert the public to maintain their distance and avoid spreading their aerosol particles that may cause the spread of viruses, including the COVID-19 virus.Originality/valueThis paper proposes an improved and augmented version of the YOLOv3 model that has been extended to perform activity recognition, such as car crash detection, human fall detection and social distancing detection. The proposed model is based on a deep learning convolutional neural network model used to detect objects in images. The model is trained using the widely used and publicly available Common Objects in Context data set. The proposed model, being an extension of YOLO, can be implemented for real-time object and activity recognition. The proposed model had higher accuracies for both large-scale and all-scale object detection. This proposed model also exceeded all the other previous methods that were compared in extending and augmenting the object detection to activity recognition. The proposed model resulted in the highest accuracy for car crash detection, fall detection and social distancing detection.

IDENTIFICATION OF FACE MASK USING CONVOLUTIONAL NEURAL NETWORK-BASED EFFICIENTNET MODEL

The highly contagious coronavirus wreaked havoc around the globe. There has been a rapid spread of the virus throughout the world. According to the World Health Organization (WHO), wearing a facemask will help keep the virus from infecting others. Consequently, many governments have adopted the solution of wearing facemasks. In this paper, we use a convolutional neural network (CNN), and a scaling method, namely, EfficientNet with Adam optimizer, for detecting face masks in real-time. A dataset including 10,000 colored images was collected from a public data platform, Kaggle, for training, and testing of the model. Image augmentation is also investigated on the dataset to improve the training, and testing accuracy. Then the binary classifier model is used to detect masks after detecting faces using single shot detection (SSD). From the experimental results, the EfficientNet model outperforms the existing CNN-based methods in terms of accuracy, efficiency, and the validation accuracy of EfficientNet models is above 99 percent. This efficient and highly accurate model can be used to detect facemask anywhere in a real time video surveillance system.

Crowd Abnormality Detection Using Optical Flow and GLCM-Based Texture Features

Detection of abnormal crowd behavior is one of the important tasks in real-time video surveillance systems for public safety in public places such as subway, shopping malls, sport complexes and various other public gatherings. Due to high density crowded scenes, the detection of crowd behavior becomes a tedious task. Hence, crowd behavior analysis becomes a hot topic of research and requires an approach with higher rate of detection. In this work, the focus is on the crowd management and present an end-to-end model for crowd behavior analysis. A feature extraction-based model using contrast, entropy, homogeneity, and uniformity features to determine the threshold on normal and abnormal activity has been proposed in this paper. The crowd behavior analysis is measured in terms of receiver operating characteristic curve (ROC) & area under curve (AUC) for UMN dataset for the proposed model and compared with other crowd analysis methods in literature to prove its worthiness. YouTube video sequences also used for anomaly detection.

Drowning Detection System using LRCN Approach

Abstract: This project provides the insights of a real-time video surveillance system capable of automatically detecting drowning incidents in a swimming pool. Drowning is the 3rd reason for the highest unintentional deaths, and that’s why it is necessary to create trustable security mechanisms. Currently, most of the swimming pool's security mechanisms include CCTV surveillance and lifeguards to help in drowning situations. But this method is not enough for huge swimming pools like in amusement parks. Nowadays, some of the security systems are using AI for drowning detection using cameras situated underwater at a fixed location and also by using floating boards having a camera mounted on the bottom side so that underwater view can be captured. But the main problems in these systems arise when the pool is crowded and vision of cameras is blocked by people. In this project, rather than using underwater cameras, we are using cameras situated on top of the swimming pool to get an upper view of the swimming pool so that entire swimming pool will be under surveillance all time. Keywords: Computer vision, Convolutional neural network, Convlstm2D, LRCN, UCF50, OpenCV.

Smart Deep Learning Based Human Behaviour Classification for Video Surveillance

Real-time video surveillance system is commonly employed to aid security professionals in preventing crimes. The use of deep learning (DL) technologies has transformed real-time video surveillance into smart video surveillance systems that automate human behavior classification. The recognition of events in the surveillance videos is considered a hot research topic in the field of computer science and it is gaining significant attention. Human action recognition (HAR) is treated as a crucial issue in several applications areas and smart video surveillance to improve the security level. The advancements of the DL models help to accomplish improved recognition performance. In this view, this paper presents a smart deep-based human behavior classification (SDL-HBC) model for real-time video surveillance. The proposed SDL-HBC model majorly aims to employ an adaptive median filtering (AMF) based pre-processing to reduce the noise content. Also, the capsule network (CapsNet) model is utilized for the extraction of feature vectors and the hyperparameter tuning of the CapsNet model takes place utilizing the Adam optimizer. Finally, the differential evolution (DE) with stacked autoencoder (SAE) model is applied for the classification of human activities in the intelligent video surveillance system. The performance validation of the SDL-HBC technique takes place using two benchmark datasets such as the KTH dataset. The experimental outcomes reported the enhanced recognition performance of the SDL-HBC technique over the recent state of art approaches with maximum accuracy of 0.9922.

A study on implementation of real-time intelligent video surveillance system based on embedded module

Conventional surveillance systems for preventing accidents and incidents do not identify 95% thereof after 22 min when one person monitors a plurality of closed circuit televisions (CCTV). To address this issue, while computer-based intelligent video surveillance systems have been studied to notify users of abnormal situations when they happen, it is not commonly used in real environment because of weakness of personal information leaks and high power consumption. To address this issue, intelligent video surveillance systems based on small devices have been studied. This paper suggests implement an intelligent video surveillance system based on embedded modules for intruder detection based on information learning, fire detection based on color and motion information, and loitering and fall detection based on human body motion. Moreover, an algorithm and an embedded module optimization method are applied for real-time processing. The implemented algorithm showed performance of 88.51% for intruder detection, 92.63% for fire detection, 80% for loitering detection and 93.54% for fall detection. The result of comparison before and after optimization about the algorithm processing time showed 50.53% of decrease, implying potential real-time driving of the intelligent image monitoring system based on embedded modules.

A real-time video surveillance system for traffic pre-events detection

In this study, a conceptual framework is proposed for the development of a video surveillance-based system for improving road safety. Based on the framework, a set of algorithms are developed which are capable of detecting various traffic pre-events from traffic videos, such as speed violation, one-way traffic, overtaking, illegal parking, and wrong drop-off location of passengers. After detecting the pre-events, an alarm will be automatically generated in the control room which helps to take precautionary measures to avoid any potential mishap on road, thereby, improving the road safety. In previous studies, a single system can handle either one or two pre-events. Whereas, in our present study, five anomalies can be detected in a single system using five different algorithms. Our study further contributes to the detection of “wrong drop-off location of passengers”. The effectiveness of the developed algorithms is demonstrated over 132 traffic videos acquired from an integrated plant in India. Some additional comparative studies for overtaking and illegal parking are done using two benchmark datasets, namely ‘CamSeq01’ and ‘ISLab-PVD’. Through an extensive study, it can be concluded that our developed algorithms are superior to some state-of-the-art algorithms in the detection of pre-events on road.

Image-Based Crowd Stability Analysis Using Improved Multi-Column Convolutional Neural Network

Crowd stability analysis is an important yet challenging task, as it is difficult to obtain the quantitative information regarding a crowd in motion, for instance, the crowd count and crowd density in a pedestrian merging area. This paper proposes a novel model that can be used to accurately analyze the crowd stability based on images obtained from a real-time video surveillance system (VSS) in dense crowd scenarios. To enhance the accuracy of the human head recognition for the crowd counting and crowd density estimation, we improve the conventional convolution-neural-network (CNN) model with more columns and merged features, obtaining a four-column convolutional neural network (4C-CNN). Using more columns with receptive fields of deferent sizes, more merged features can be learned, to be adaptive to variations in pedestrian head size due to image resolution. Furthermore, the crowd density of different areas is calculated with image rectification against the perspective distortion. By utilizing the stability criterion based on crowd density, we propose a crowd stability analysis model (CSAM) with the capability of quantitative computation dynamically. The results of extensive experiments performed on public datasets indicate that this improved CNN model exhibits a better performance for crowd counting than the typical multi-column CNN models. In addition, the experiment results pertaining to Shanghai Hongqiao Railway Station demonstrate the effectiveness of the crowd stability analysis model. Thus, this integrated approach can simultaneously conduct the processes of crowd counting, image rectification, density map calculation and crowd stability analysis by using dense crowd images from a VSS.

Hard Disk Drive Failure Prediction for Mobile Edge Computing Based on an LSTM Recurrent Neural Network

With the increase in intelligence applications and services, like real-time video surveillance systems, mobile edge computing, and Internet of things (IoT), technology is greatly involved in our daily life. However, the reliability of these systems cannot be always guaranteed due to the hard disk drive (HDD) failures of edge nodes. Specifically, a lot of read/write operations and hazard edge environments make the maintenance work even harder. HDD failure prediction is one of the scalable and low-overhead proactive fault tolerant approaches to improve device reliability. In this paper, we propose an LSTM recurrent neural network-based HDD failure prediction model, which leverages the long temporal dependence feature of the drive health data to improve prediction efficiency. In addition, we design a new health degree evaluation method, which stores current health details and deterioration. The comprehensive experiments on two real-world hard drive datasets demonstrate that the proposed approach achieves a good prediction accuracy with low overhead.

Design and PoC Implementation of Mmwave-Based Offloading-Enabled UAV Surveillance System

Among UAV applications, real-time high-resolution video surveillance systems using unmanned aerial vehicles (UAVs) especially gain our research interests, because high mobilities of UAVs and high qualities of videos can dramatically improve the surveillance performance and evolve the system, especially for artificial intelligence (AI)-based monitoring systems. However, it is challenging to transmit high-resolution videos to ground station in real-time and also impossible to perform AI detection based on these videos on UAVs. To address the challenges, we propose to take full advantages of mmWave communications and to apply it into UAV-based surveillance systems to transmit ultra-high-quality and ultra-high-resolution uncompressed 4K videos. To increase energy efficiency of UAVs, we propose to offload all computations via mmWave from UAVs to ground station. Moreover, in addition to design of system architecture, proof-of-concept (PoC) prototype hardware of the proposed system is practically developed to demonstrate technical feasibilities of mmWave communication on UAVs and the enhancements to practical surveillance applications. The practical issues in the implementation and the necessary hardware/software applied in the prototype system are explained in detail in this paper. An outdoor PoC experiment is conducted, and the experimental results are consistent with design, so that the effectiveness of the system is confirmed.

Detection and Location of Safety Protective Wear in Power Substation Operation Using Wear-Enhanced YOLOv3 Algorithm

Wearing personal safety protective equipment (PSPE) plays a key role in reducing electrical injuries to electrical workers. However, substation employees often ignore this regulation due to lack of safety awareness and discomfortable feeling of wearing PSPE. Therefore, it is necessary to develop a detection algorithm for PSPE and workers to build real-time video surveillance systems in power substations. In this paper, a wear-enhanced YOLOv3 method for real-time detection of PSPE and substation workers is proposed. The gamma correction is first applied as the preprocessing method to highlight the details of the operators and data augmentation is performed. Next, K-means++ algorithm replaces K-means in wear-enhanced YOLOv3 method to derive the most suitable prior bounding box size and lift the detection speed. Then, the proposed method can be quickly and effectively trained based on transfer learning. Finally, extensive experiments are carried out on a dataset of images about usage of safety helmets, insulating gloves and boots. Using the proposed method, the mean average precision is improved by over 2% and the frames per second is the highest compared with other typical object detection methods, which illustrates the effectiveness of the wear-enhanced YOLOv3 method for PSPE and workers detection.

Real-time video fire/smoke detection based on CNN in antifire surveillance systems

This work presents a real-time video-based fire and smoke detection using YOLOv2 Convolutional Neural Network (CNN) in antifire surveillance systems. YOLOv2 is designed with light-weight neural network architecture to account the requirements of embedded platforms. The training stage is processed off-line with indoor and outdoor fire and smoke image sets in different indoor and outdoor scenarios. Ground truth labeler app is used to generate the ground truth data from the training set. The trained model was tested and compared to the other state-of-the-art methods. We used a large scale of fire/smoke and negative videos in different environments, both indoor (e.g., a railway carriage, container, bus wagon, or home/office) or outdoor (e.g., storage or parking area). YOLOv2 is a better option compared to the other approaches for real-time fire/smoke detection. This work has been deployed in a low-cost embedded device (Jetson Nano), which is composed of a single, fixed camera per scene, working in the visible spectral range. There are not specific requirements for the video camera. Hence, when the proposed solution is applied for safety on-board vehicles, or in transport infrastructures, or smart cities, the camera installed in closed-circuit television surveillance systems can be reused. The achieved experimental results show that the proposed solution is suitable for creating a smart and real-time video-surveillance system for fire/smoke detection.

Analytical Modelling of Video Surveillance System Along Forest Railway Tracks

An analytical model has been developed by the authors for determining the angular alignment, position, length related parameters of a proposed real-time video surveillance system to be installed along railway tracks or highways passing through open or moderately dense or very dense forest regions. The proposed system is capable of providing 24 × 7 real-time surveillance of the railway tracks or highways and its surroundings in order to avoid accidents caused by animal-train or animal-vehicle collisions.

Real-time smart video surveillance to manage safety: A case study of a transport mega-project

There is a tendency for accidents and even fatalities to arise when people enter hazardous work areas during the construction of projects in urban areas. A limited amount of research has been devoted to developing vision-based proximity warning systems that can determine when people enter a hazardous area automatically. Such systems, however, are unable to identify specific hazards and the status of a piece of plant (e.g., excavator) in real-time. In this paper, we address this limitation and develop a real-time smart video surveillance system that can detect people and the status of plant (i.e. moving or stationary) in a hazardous area. The application of this approach is demonstrated during the construction of a mega-project, the Wuhan Rail Transit System in China. We reveal that our combination of computer vision and deep learning can accurately recognize people in a hazardous work area in real-time during the construction of transport projects. Our developed systems can provide instant feedback concerning unsafe behavior and thus enable appropriate actions to be put in place to prevent their re-occurrence.

SAT: Single-Shot Adversarial Tracker

Deep learning-based tracking methods have shown favorable performance on multiple benchmarks. However, most of these methods are not designed for real-time video surveillance systems due to the complex online optimization process. In this article, we propose a single-shot adversarial tracker (SAT) to efficiently locate objects of interest in surveillance videos. Specifically, we propose a lightweight convolutional neural network-based generator, which fuses multilayer feature maps to accurately generate the target probability map (TPM) for tracking. To more effectively train the generator, an adversarial learning framework is presented. During the online tracking stage, the learned TPM generator can be directly employed to generate the target probability map corresponding to the searching region in a single shot. The proposed SAT can lead to the average tracking speed of 212 FPS on a single GPU, while still achieving the favorable performance on several popular benchmarks. Furthermore, we also present a variant of SAT by considering both scale estimation and online updating in SAT, which achieves better accuracy than SAT while still maintaining very fast tracking speed (i.e., exceeding 100 FPS).

Embedded Real-Time Video Surveillance System based on Multi-Sensor and Visual Tracking

Embedded Real-Time Video Surveillance System based on Multi-Sensor and Visual Tracking

Multiperiod Network Rate Allocation With End-to-End Delay Constraints

QoS-aware networking applications such as real-time streaming and video surveillance systems require nearly fixed average end-to-end delay over long periods to communicate efficiently, although may tolerate some delay variations in short periods. This variability exhibits complex dynamics that makes rate control of such applications a formidable task. This paper addresses rate allocation for heterogeneous QoS-aware applications that preserves the long-term end-to-end delay constraint while seeking the maximum network utility cumulated over a fixed time interval. To capture the temporal dynamics of sources, we incorporate a novel time-coupling constraint in which delay sensitivity of sources is considered such that a certain end-to-end average delay for each source over a prespecified time interval is satisfied. We propose an algorithm, as a dual-based solution, which allocates source rates for the next time interval in a distributed fashion, given the knowledge of network parameters in advance. Also, we extend the algorithm to the case that the problem data is not known fully in advance to capture more realistic scenarios. Through numerical experiments, we show that our proposed algorithm attains higher average link utilization and a wider range of feasible scenarios in comparison with the best, to our knowledge, rate control schemes that may guarantee such constraints on delay.

Live Video Streaming from Remote Location Using Raspberry Pi

Objectives: Through this paper, design of a reliable real-time (Live) video streaming and surveillance system is proposed, which can be broadcasted over a Local-Area Network (LAN/ WLAN) with remote access. Methods/Statistical Analysis: Proposed system design features Raspberry Pi along with night-vision camera, programmed using Python programming language. Raspberry Pi’s IP address is fetched and VNC server is setup to create virtual desktop on host device for remoteuser. Host device is a Personal Computer/Smartphone connected to same LAN/ WLAN on which Raspberry Pi is connected. VLC media player serves as source for live streaming on host device running a VNC viewer. Findings: Previously, live video surveillance meant massive machinery, restricted to larger spaces and usually involved Closed-circuit television cameras. The technological advancements, ease of access and usability in Latter-day gadgets brought forth easier connectivity, bridging the gap between virtual and the real worlds. This makes dealing with indispensable concepts like security and surveillance much convenient, even for a remote user. Keeping in mind all these concepts, an effective; simple yet powerful design for a live video streaming and surveillance system has been proposed. For effective implementation of the design, Raspberry Pi 3 Model B along with a Raspberry Pi NoIR camera V2 module has been used and VLC media player acts as the streaming medium for the live video broadcast, initiated by the code written in Python. VNC server and viewer are used for remote connections over the Local-Area Network. And since a night-vision camera is being utilized, it allows the user to keep an eye on the area even in dark and low light conditions with ease. Application/Improvements: The system proposed offers numerous amounts of applications due to its versatility. Through this design, a hassle-free live video streaming/surveillance may be effortlessly availed even by an average user. Keywords: Live Streaming, Python, Raspberry Pi, Video Surveillance, Virtual Network Computing, VLC Media Player

Real Time Video Surveillance System for Fire and Smoke Detection Based on Wavelet Transform.

Fire detection processing technique for Video Surveillance Systems (VSS) has attracted the interest of a lot of researchers because of its crucial value in various applications in our daily life. In this research work, a real time video surveillance system for fire detection based on wavelet transforms is proposed. It aims to design and implement fire detection in a spatial and wavelet domain domain. The input video is first extracted into frames and then the motion detection algorithm for the change detection is applied to separate moving objects from the static objects. Test results of the proposed fire and smoke detection methods at different distance from one to ten meters indicate that the accuracy of the system for frames with performing contour algorithm (average accuracy for (1-10) meter is equal to (99.46%). On the other hand, in a frequency domain the system has a better performance for detecting energy of fire than a smoke.

Efficient Human Motion Detection with Adaptive Background for Vision-Based Security System

Motion detection is very important in video surveillance system especially for video compression, human detection and behaviour analysis. Various approaches have been used for detecting motion in a continuous video stream but for real-time video surveillance system, we need a motion detection that can provide accurate detection even in non-static background regardless of surroundings (outdoor or indoor), object speed and size, robust to camera noisy pixels or sudden change in light intensity. This is very important to ensure that the security of a monitored parameter or area is not compromised. In this paper, we propose a method for human motion detection which employs adaptive background subtraction, camera noise reduction and white pixel counts threshold for real-time video streams.