Intelligent Visual Surveillance System Research Articles

Suspicious Activity Classification in Classrooms using Deep Learning

Video processing is attracting the attention of both research and industry. The existence of intelligent surveillance cameras with high processing power has paved the way for designing intelligent visual surveillance systems. Along with analyzing video for information recovery, it is nowadays used to analyze live surveillance video to detect activities. These systems are implemented in real time. The proposed work's goal is to create a method that can examine and discover suspicious behaviors in the lecture room environment. Video analytics offers the most efficient answer because it enables pointing an occasion and retrieves applicable statistics from the video recorded. The method aims to identify suspicious activities like fighting, sleeping, looking elsewhere, eating, etc. that the students might be doing. The proposed method involves breaking a video input into frames and converting it into image data because the model has been trained on images collected from the internet. Several models were tested and experimented with, including efficientnet_b2, spnasnet_100, efficientnet_b3, and mobilenetv3_large_100. Parameters such as the Learning Rate were optimized to find out the best method and create a system with the best results.

URRNet: A Unified Relational Reasoning Network for Vehicle Re-Identification

With the continuous improvement and optimization of security monitoring networks, vehicle Re-Identification (Re-ID) becomes an emerging key technology in the development of intelligent visual surveillance systems. Due to the influence of viewpoint variation and fine-grained differences, vehicle Re-ID is still a research topic worth investigating. To alleviate above problems, a novel end-to-end framework named Unified Relational Reasoning Network (URRNet) is proposed in this paper, which integrates global features with local features to obtain better recognition accuracy. For the proposed framework, to understand the overall semantics of the image, an algorithm based on the global feature graph-structure learning is designed. The pixel-level feature maps are transformed to the node features of graph in the interactive space by projection, then graph reasoning is performed by using the graph convolutional network to improve the representation of global features. Moreover, an algorithm based on multi-scale local feature relational reasoning is designed. Using keypoint and viewpoint to obtain the multi-scale partial characteristics of the vehicle, and the vehicle multi-view features are learned from the single-view vehicle images through relational reasoning and attention mechanism. The two algorithms are combined to obtain the overall model, which not only preserves the details of the vehicle, but also effectively solves the problem of viewpoint variation. Comprehensive experimental results on two public datasets (VeRi-776 and VehicleID) indicate that the proposed URRNet can practically improve the model's representation ability and generalization ability, which is comparable to the state-of-the-art vehicle Re-ID methods.

Towards Building a Visual Behaviour Analysis Pipeline for Suicide Detection and Prevention.

Understanding human behaviours through video analysis has seen significant research progress in recent years with the advancement of deep learning. This topic is of great importance to the next generation of intelligent visual surveillance systems which are capable of real-time detection and analysis of human behaviours. One important application is to automatically monitor and detect individuals who are in crisis at suicide hotspots to facilitate early intervention and prevention. However, there is still a significant gap between research in human action recognition and visual video processing in general, and their application to monitor hotspots for suicide prevention. While complex backgrounds, non-rigid movements of pedestrians and limitations of surveillance cameras and multi-task requirements for a surveillance system all pose challenges to the development of such systems, a further challenge is the detection of crisis behaviours before a suicide attempt is made, and there is a paucity of datasets in this area due to privacy and confidentiality issues. Most relevant research only applies to detecting suicides such as hangings or jumps from bridges, providing no potential for early prevention. In this research, these problems are addressed by proposing a new modular design for an intelligent visual processing pipeline that is capable of pedestrian detection, tracking, pose estimation and recognition of both normal actions and high risk behavioural cues that are important indicators of a suicide attempt. Specifically, based on the key finding that human body gestures can be used for the detection of social signals that potentially precede a suicide attempt, a new 2D skeleton-based action recognition algorithm is proposed. By using a two-branch network that takes advantage of three types of skeleton-based features extracted from a sequence of frames and a stacked LSTM structure, the model predicts the action label at each time step. It achieved good performance on both the public dataset JHMDB and a smaller private CCTV footage collection on action recognition. Moreover, a logical layer, which uses knowledge from a human coding study to recognise pre-suicide behaviour indicators, has been built on top of the action recognition module to compensate for the small dataset size. It enables complex behaviour patterns to be recognised even from smaller datasets. The whole pipeline has been tested in a real-world application of suicide prevention using simulated footage from a surveillance system installed at a suicide hotspot, and preliminary results confirm its effectiveness at capturing crisis behaviour indicators for early detection and prevention of suicide.

Suspicious activity recognition for monitoring cheating in exams

Video processing is getting special attention from research and industries. The existence of smart surveillance cameras with high processing power has opened the for making it conceivable to design intelligent visual surveillance systems. Know a day it is very possible to assure invigilators safety during the examination period. This work aims to distinguish the suspicious activities of students during the exam for surveillance examination halls. For this, a 63 layers deep CNN model is suggested and named "L4-BranchedActionNet". The suggested CNN structure is centered on the alteration of VGG-16 with added four blanched. The developed framework is initially turned into a pre-trained framework by using the SoftMax function to train it on the CUI-EXAM dataset. The dataset for detecting suspicious activity is subsequently sent to this pre-trained algorithm for feature extraction. Feature subset optimization is applied to the deep features that have been obtained. These extracted features are first entropy coded, and then an ant colony system (ACS) is used to optimize the entropy-based coded features. The configured features are then input into a variety of classification models based on SVM and KNN. With a performance of 0.9299 in terms of accuracy, the cubic SVM gets the greatest efficiency scores. The suggested model was further tested on the CIFAR-100 dataset, and it was shown to be accurate to the tune of 0.89796. The result indicates the suggested frameworks soundness.

Vehicle and Person Re-Identification With Support Neighbor Loss.

One of the key tasks for an intelligent visual surveillance system is to automatically re-identify objects of interest, e.g., persons or vehicles, from nonoverlapping camera views. This demand incurs the vast investigation of person re-identification (re-ID) and vehicle re-ID techniques, especially those deep learning-based ones. While most recent algorithms focus on designing new convolutional neural networks, less attention is paid to the loss functions, which are of vital roles as well. Triplet loss and softmax loss are the two losses that are extensively used, both of which, however, have limitations. Triplet loss optimizes the model to produce features with which samples from the same class have higher similarity than those from different classes. The problem of triplet loss is that the number of triplets to be constructed grows cubically with training samples, which causes scalability issue, unstable performance, and slow convergence. Softmax loss has favorable scalable property and is widely used for large-scale classification problems. However, since Softmax loss only aims to separate well training classes, its performance for re-ID tasks is not desirable because the model is tested to measure the similarity of samples from unseen classes. We propose the support neighbor (SN) loss, which avoids the limitations of the abovementioned two losses. Unlike triplet loss that is calculated based on triplets, SN loss is derived from K -nearest neighbors (SNs) of anchor samples. The SNs of an anchor are unique, containing more valuable contextual information and neighborhood structure of the anchor, and thus contribute to more stable performance and reliable embedding from image space to feature space. Based on the SNs, a softmax-like separation term and a squeeze term are proposed, which encourage interclass separation and intraclass compactness, respectively. Experiments show that SN loss surpasses triplet and softmax losses with the same backbone network and reaches the state-of-the-art performance for both person and vehicle re-ID using a ResNet50 backbone when combined with training tricks.

Person Reidentification Based on Automotive Radar Point Clouds

Person reidentification (ReID) systems play a key role in intelligent visual surveillance systems and have widespread applications, for example, in public security. Usually, person ReID systems can identify a person with cameras. In this article, we focus on the relatively unexplored area of using low-cost automotive radar for the person ReID problem. Unlike the radar-based person identification, person ReID has some characteristics, such as the uncooperative scenes and the long-term robustness. Therefore, we design a new deep learning network to extract spatiotemporal information from 4-D radar point clouds. We also build a data set of radar point clouds collected from the real-world person ReID scenarios. The evaluation result shows that our method achieves 91% CMC-1 accuracy on the ReID task. Besides, for the person identification task, our method also achieves accuracies of 98% and 91% for 15 and 40 individuals, respectively. In addition, we discuss the potential of using radar for person ReID problems and intuitively explain the new method’s performance. Finally, we analyze the robustness and the influence of different parameters on the method and the contributions of different modules to the network model. The results of our experiment indicate that radar-based ReID not only preserves privacy but also outperforms camera-based ReID in some cases, such as in low-light environments or with substantial clothing changes.

An Energy-Efficient Accelerator for Rain Removal Based on Convolutional Neural Network

Rain removal is an important operation in intelligent visual surveillance systems. Many rain removal algorithms based on convolution neural networks (CNNs) are rarely deployed on resource constrained devices. One of the limiting factors is that memory access leads to high energy consumption. To reduce memory access during computation, previous works usually use a fixed computation pattern for different layers in CNN. For different and massive input and output feature maps, fixed computation pattern would lower the power efficiency. Thus, we propose a reconfigurable architecture to support different convolution mapping method. We use hybrid data reuse pattern to reduce energy consumption by 2.4-5.9 times over fixed computation pattern. The hardware is synthesized in the area of 9.83 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> at TSMC 65 nm technology and can restore a 1280 ×720 real world image in 0.57 s which achieves 56.1 × and 2.1 × speed-up comparing to CPU and GPU implementations. The simulation results show that the power efficiency is 287.7 GOPS/W running rain removal network.

Suspicious Activity Recognition Using Proposed Deep L4-Branched-Actionnet With Entropy Coded Ant Colony System Optimization

Intelligent visual surveillance systems are attracting much attention from research and industry. The invention of smart surveillance cameras with greater processing power has now been the leading stakeholder, making it conceivable to design intelligent visual surveillance systems. It is possible to assure the safety of people in both homes and public places. This work aims to distinguish the suspicious activities for surveillance environments. For this, a 63 layers deep CNN model is suggested and named “L4-Branched-ActionNet”. The suggested CNN structure is centered on the alteration of AlexNet with added four blanched sub-structures. The developed framework is first transformed into a pre-trained framework by conducting its training on an object detection dataset called CIFAR-100 with the SoftMax function. The dataset for suspicious activity recognition is then forwarded to this pretrained model for feature acquisition. The acquired deep features are subjected to feature subset optimization. These extracted features are first coded by applying entropy and then an ant colony system (ACS) is utilized on the entropy-based coded features for optimization. The configured features are then fed into numerous SVM and KNN based classification models. The cubic SVM has the highest efficiency scores, with a performance of 0.9924 in order of accuracy. The proposed model is also validated on the Weizmann action dataset and attained an accuracy of 0.9796. The successful findings indicate the suggested work’s soundness.

A Survey on Deep Learning-Based Person Re-Identification Systems

Person re-identification systems (person Re-ID) have recently gained more attention between computer vision researchers. They are playing a key role in intelligent visual surveillance systems and have widespread applications like applications for public security. The person Re-ID systems can identify if a person has been seen by a non-overlapping camera over large camera network in an unconstrained environment. It is a challenging issue since a person appears differently under different camera views and faces many challenges such as pose variation, occlusion and illumination changes. Many methods had been introduced for generating handcrafted features aimed to handle the person Re-ID problem. In recent years, many studies have started to apply deep learning methods to enhance the person Re-ID performance due the deep learning yielded significant results in computer vision issues. Therefore, this paper is a survey of the recent studies that proposed to improve the person Re-ID systems using deep learning. The public datasets that are used for evaluating these systems are discussed. Finally, the paper addresses future directions and current issues that must be considered toward improving the person Re-ID systems.

Deep Fusion Feature Presentations for Nonaligned Person Re-Identification

Person re-identification aims to retrieve the pedestrian across different cameras. It is still a challenging task for the intelligent visual surveillance system because of similar appearances, camera shooting angles, scene illumination, and pedestrian pose. In this paper, we propose a novel two-stream network named spatial segmentation network that learns both the global and local features in a unified framework for nonaligned person re-identification. One stream focuses on spatial feature learning using global adaptive average pooling in deep convolutional neural networks. Another stream is utilized to learn the fine local features by adopting horizontal average pooling without division that depends on the pose predictor. To assess the importance ranking of all features, we also obtain the performance of every part feature and global features. Our evaluation of the proposed method on Market-1501 acquires 94.51% Rank-1 and 90.78% mAP, that on DukeMTMC-re-ID acquires 87.52% Rank-1 and 84.82% mAP, and that on CHUK03-detected acquires 69.71% Rank-1 and 71.67% mAP; these findings verify the state-of-the-art performance of the proposed method.

РАЗРАБОТКА СРЕДСТВ АГЕНТНОГО ЛОГИЧЕСКОГО ПРОГРАММИРОВАНИЯ ДЛЯ МНОГОКАНАЛЬНОГО ИНТЕЛЛЕКТУАЛЬНОГО ВИДЕОНАБЛЮДЕНИЯ

Experimental means developed in the Actor Prolog parallel object-oriented logic language for implementation of heterogeneous multichannel intelligent visual surveillance systems are considered. These means are examined by the instance of a logic program for permanent monitoring of people's body parts temperature in the area of visual surveillance. The logic program implements a fusion of heterogeneous data acquired by two devices: (1) 3D coordinates of human body are measured using a time-of-flight (ToF) camera; (2) 3D coordinates of human body skeleton are computed on the base of 3D coordinates of the body; (3) a thermal video is acquired using a thermal imaging camera. Special built-in classes are developed in the Actor Prolog language for the acquisition and analysis of 3D and 2D video data: the Kinect and KinectBuffer classes implement interaction between the logic program and the ToF camera of the Kinect-2 device; the BufferedScene class implements storing and transferring 3D images; the Canvas3D class implements 3D graphics based on the Java3D open source library. In the considered example, the thermal video is projected to the 3D surface of the human body; then the temperature of the human body is projected to the vertices and edges of the skeleton. A special logical agent (i.e., the logic program written in Actor Prolog) implements these operations in real-time and transfers the data to another logical agent. The latter agent implements a time average of the temperature of the human skeletons and displays colored 3D images of the skeletons; the average temperature of the vertices and edges of the skeletons is depicted by colors. A method of remote predicate calls is used for the interaction of the logical agents; this method was developed and implemented in Actor Prolog to support the agent logic programming paradigm. The logic programming means under consideration are developed for the purpose of the implementation of logical analysis of video scene semantics in the intelligent visual surveillance systems. © 2018 Radioelektronika, Nanosistemy, Informacionnye Tehnologii. All rights reserved.

Moving Object Detection Using Scanning Camera on a High-Precision Intelligent Holder.

During the process of moving object detection in an intelligent visual surveillance system, a scenario with complex background is sure to appear. The traditional methods, such as “frame difference” and “optical flow”, may not able to deal with the problem very well. In such scenarios, we use a modified algorithm to do the background modeling work. In this paper, we use edge detection to get an edge difference image just to enhance the ability of resistance illumination variation. Then we use a “multi-block temporal-analyzing LBP (Local Binary Pattern)” algorithm to do the segmentation. In the end, a connected component is used to locate the object. We also produce a hardware platform, the core of which consists of the DSP (Digital Signal Processor) and FPGA (Field Programmable Gate Array) platforms and the high-precision intelligent holder.

Automatic Detection of Events and Tracking of Moving Objects in Video Sequences

Digital video is being used widely in a variety of applications such as surveillance and security. Big amount of video in surveillance and security requires systems capable to process video automatically to detect events and track moving objects to alleviate the load on humans and enable preventive actions when events are detected [4]. our paper focuses to develop an intelligent visual surveillance system to replace the traditional passive video surveillance that is proving ineffective as the number of cameras exceeds the capability of human operators to monitor them, and it is able to track objects within a maximum solid angle speed which is measured at about 0.3 to 0.2 radian per second, further it also depends on the complexity of the system and the processor speed as well.

Real-Time Automatic Obstacle Detection method for Traffic Surveillance in Urban Traffic

Obstacle detection in urban traffic is a hot topic in intelligent visual surveillance systems. In this paper, a real-time automatic obstacle recognition method based on computer vision technology is presented. The proposed method aims at detecting and recognizing the road obstacles such as abandoned objects, accident vehicles and illegally parked vehicles, which can prevent the traffic accident effectively. In the method, the target images are captured by a visible image sensor firstly. In order to avoid the static objects disappearing from foreground in short time when using GMM (Gaussian Mixture Model), background is built and foreground objects are extracted by the proposed algorithm SUOG (Selective Updating of GMM). Relative object speed is used to detect the static obstacles, and FROI (Flushed Region of Interest) algorithm based on the concept of connected domain, is presented to eliminate noises outside road and improve real-time capability. At last, a classification method of adaptive interested region based on HOG and SVM, and a new recognition algorithm of accident vehicles based on multi-feature fusion are proposed to classify the road obstacles. Experiments indicate that the detection rate of the proposed obstacle detection method is up to 96 % in urban road traffic. Through experiment, it is shown that the developed obstacle detection method has low computational complexity, and can fulfill the requirement of real-time applications, and it is correct and effective.

Application of Object Detection and Tracking Techniques for Unmanned Aerial Vehicles

In this research, the information captured by Unmanned Aerial Vehicles (UAVs) are eminently utilized in detecting and tracking moving objects which pose a primary security threat against the United States southern border. Illegal trespassing and border encroachment by immigrants is a huge predicament against the United States border security force and the Department of Homeland Security. It becomes insurmountable to warranty suspicious behaviour, monitoring by human operators for long periods of time, due to the massive amount of data involved. The main objective of this research is to assist the human operators, by implementing intelligent visual surveillance systems which help in detecting and tracking suspicious or unusual events in the video sequence. The visual surveillance system requires fast and robust methods of detecting and tracking moving objects. In this research, we have investigated methods for detecting and tracking objects from UAVs. Moving objects were detected using adaptive background subtraction technique successfully and these detected objects were tracked by using Lucas-Kanade optical flow tracking, Continuously Adaptive Mean-Shift tracking based techniques. The simulation results show the efficacy of these techniques in detecting and tracking moving objects in the video sequences acquired by the UAV.

Multiple pedestrian tracking based on couple-states Markov chain with semantic topic learning for video surveillance

Robust, accurate and efficient pedestrian tracking is a critical task in intelligent visual surveillance systems and robotic vision applications. Unfortunately, tracking in realistic scenarios is not easy and may fail due to the challenges of partial occlusion, viewpoint changes and cluttered background. To overcome these challenges, different tracking strategies have been proposed to model the tracking process as a first-order temporal Markov chain. It has been well-known that target appearance modeling approaches play essential function in this process, but most these approaches only represent the object characteristics in the pixel/texture level instead of investigating the latent information in the semantic understanding level. Therefore, the obtained optimum state based on the texture similarity may not be as same as observed by human vision system. To resolve this limitation, in this paper, we proposed a multiple pedestrian tracking algorithm based on couple-states analysis, the hidden state is used to obtain the estimated observations during the Markov chain transition process, and the latent state is used to find the semantic information about each observation. By maximizing the likelihood probability of the couple states for each estimation, the optimum state of target can be found more accurately, and error accumulation can also be effectively decreased during tracking. The performance of the proposed tracking has been verified on different benchmark surveillance video database, the results showed that the proposed tracking is able to track multiple pedestrians more accurately in variety of challenge scenarios when compared with other state-of-art multiple pedestrian tracking approaches.

A Survey on Moving Target Tracking in the Intelligent Visual Monitoring System

Moving target tracking is the key part of intelligent visual surveillance system. Among the various tracking algorithms, the Beysian tracking algorithms and the kernel tracking algorithm are two algorithms that frequently used. The Beysian tracking algorithms mainly conclude Kalman filtering algorithm, extended Kalman filtering algorithm and particle filtering algorithm. Mean Shift is the most representative algorithm of the kernel target tracking. In this survey, the status and development of target tracking algorithms has been studied more extensively with providing a few examples of modified tracking algorithms. Then a comparison was presented based on the limitations and scope of applications. Finally, the paper showed further research prospects of moving target tracking are introduced.

Scene-adaptive accurate and fast vertical crowd counting via joint using depth and color information

Reliable and real-time crowd counting is one of the most important tasks in intelligent visual surveillance systems. Most previous works only count passing people based on color information. Owing to the restrictions of color information influences themselves for multimedia processing, they will be affected inevitably by the unpredictable complex environments (e.g. illumination, occlusion, and shadow). To overcome this bottleneck, we propose a new algorithm by multimodal joint information processing for crowd counting. In our method, we use color and depth information together with a ordinary depth camera (e.g. Microsoft Kinect). Specifically, we first detect each head of the passing or still person in the surveillance region with adaptive modulation ability to varying scenes on depth information. Then, we track and count each detected head on color information. The characteristic advantage of our algorithm is that it is scene adaptive, which means the algorithm can be applied into all kinds of different scenes directly without additional conditions. Based on the proposed approach, we have built a practical system for robust and fast crowd counting facing complicated scenes. Extensive experimental results show the effectiveness of our proposed method.

Learning Patterns of Motion Trajectories Using Real-Time Tracking

The understanding and description of behaviors for road vehicles is a hot topic of intelligent visual surveillance system. Trajectory analysis is one of the basic problems in behavior understanding, from which anomalies can be detected and also accidents can be predicted. In this paper, we proposed a hierarchical self-organizing neural network model to learn trajectory distribution pattern and a probability model for accident recognition. Sample data including motion trajectories are first get by real-time vehicle tracking. The self-organizing neural network algorithm is then applied to learn activity patterns from the sample trajectories. Using the learned patterns, we consider anomaly detection as well as object behavior prediction. Experiments in actual road scene show the effectiveness of the proposed algorithm.

Special section on advances in intelligent visual surveillance systems

Special section on advances in intelligent visual surveillance systems