Distributed Surveillance System Research Articles

Multi-UAV Coverage Path Planning: A Distributed Online Cooperation Method

Coverage path planning (CPP) for unmanned aerial vehicles (UAVs) plays a significant role in intelligent distributed surveillance systems. However, due to poor cooperation, most existing CPP methods may cause strongly overlapped trajectories, missing areas, or even collisions in uncertain and complex environments, leading to long task completion time and low coverage efficiency. To this end, in this paper we propose a novel multi-UAV distributed online cooperation (MDOC) CPP method that aims to minimize task completion time. Moreover, this method allows UAVs to quickly respond to unknown obstacles and complex emergencies, such as UAV breakdown or communication interruption. To establish close cooperation between UAVs, we propose an efficient environmental information map (EI-map) fusion technique that enables them to obtain global exploration in real-time in a cooperative manner. Then we innovatively develop a distributed cooperative deep Q-learning (DCDQN) algorithm to obtain UAVs' coverage paths online that are determined by minimizing task time and avoiding overlaps, missing areas, and collisions. Specifically, attributing to the fused EI-map, we expand the state space of DCDQN to collect sufficient observations and design a novel cooperative learning pattern to efficiently plan the path for global optimization. Simulation results show that our method outperforms the state-of-the-art in task completion time and coverage efficiency, especially in uncertain and complex environments. In addition, we validate that our method can efficiently complete full coverage even in emergencies.

Automatic extraction channel of space debris based on wide-field surveillance system.

In the past few years, the increasing amount of space debris has triggered the demand for distributed surveillance systems. Long exposure time can effectively improve the target detection capability of the wide-area surveillance system. Problems that also cause difficulties in space-target detection include large amounts of data, countless star points, and discontinuous or nonlinear targets. In response to these problems, this paper proposes a high-precision space-target detection and tracking pipeline that aims to automatically detect debris data in space. First, a guided filter is used to effectively remove the stars and noise, then Hough transform is used to detect space debris, and finally Kalman filter is applied to track the space debris target. All experimental images are from Jilin Observatory, and the telescope is in star-tracking mode. Our method is practical and effective. The results show that the proposed automatic extraction channel of space debris can accurately detect and track space targets in a complex background.

Federated Dynamic Graph Neural Networks with Secure Aggregation for Video-based Distributed Surveillance

Distributed surveillance systems have the ability to detect, track, and snapshot objects moving around in a certain space. The systems generate video data from multiple personal devices or street cameras. Intelligent video-analysis models are needed to learn dynamic representation of the objects for detection and tracking. Can we exploit the structural and dynamic information without storing the spatiotemporal video data at a central server that leads to a violation of user privacy? In this work, we introduce Federated Dynamic Graph Neural Network (Feddy), a distributed and secured framework to learn the object representations from graph sequences: (1) It aggregates structural information from nearby objects in the current graph as well as dynamic information from those in the previous graph. It uses a self-supervised loss of predicting the trajectories of objects. (2) It is trained in a federated learning manner. The centrally located server sends the model to user devices. Local models on the respective user devices learn and periodically send their learning to the central server without ever exposing the user’s data to server. (3) Studies showed that the aggregated parameters could be inspected though decrypted when broadcast to clients for model synchronizing, after the server performed a weighted average. We design an appropriate aggregation mechanism of secure aggregation primitives that can protect the security and privacy in federated learning with scalability. Experiments on four video camera datasets as well as simulation demonstrate that Feddy achieves great effectiveness and security.

Smart Surveillance System Real-Time Multi-Person Multi-Camera Tracking at the Edge

Abstract: Nowadays, new Artificial Intelligence (AI) and Deep Learning based processing methods are replacing traditional computer vision algorithms. On the other hand, the rise of the Internet of Things (IoT) and edge computing, has led to many research works that propose distributed video-surveillance systems based on this notion. Usually, the advanced systems process massive volumes of data in different computing facilities. Instead, this paper presents a system that incorporates AI algorithms into low-power embedded devices. The computer vision technique, which is commonly used in surveillance applications, is designed to identify, count, and monitor people's movements in the area. A distributed camera system is required for this application. The proposed AI system detects people in the monitored area using a MobileNet-SSD architecture. This algorithm can keep track of people in the surveillance providing the number of people present in the frame. The proposed framework is both privacy-aware and scalable supporting a processing pipeline on the edge consisting of person detection, tracking and robust person re-identification. The expected results show the usefulness of deploying this smart camera node throughout a distributed surveillance system. Keywords: Edge Analytics, Person detection, Person re-identification, Deep learning, embedded systems, artificial intelligence

Smart Video Surveillance System Based on Edge Computing.

New processing methods based on artificial intelligence (AI) and deep learning are replacing traditional computer vision algorithms. The more advanced systems can process huge amounts of data in large computing facilities. In contrast, this paper presents a smart video surveillance system executing AI algorithms in low power consumption embedded devices. The computer vision algorithm, typical for surveillance applications, aims to detect, count and track people’s movements in the area. This application requires a distributed smart camera system. The proposed AI application allows detecting people in the surveillance area using a MobileNet-SSD architecture. In addition, using a robust Kalman filter bank, the algorithm can keep track of people in the video also providing people counting information. The detection results are excellent considering the constraints imposed on the process. The selected architecture for the edge node is based on a UpSquared2 device that includes a vision processor unit (VPU) capable of accelerating the AI CNN inference. The results section provides information about the image processing time when multiple video cameras are connected to the same edge node, people detection precision and recall curves, and the energy consumption of the system. The discussion of results shows the usefulness of deploying this smart camera node throughout a distributed surveillance system.

Ghost protocol – Snapchat as a method of surveillance

Social Networking Sites (SNS) are a rich source of open-source intelligence in the form of publicly available images, video and text. Privacy concerns have led to many sites removing metadata attached to multimedia on upload, effectively losing information of interest from intelligence perspectives. In this paper, we describe the exploitation of Snapchat's Snap Map (a web based portal to access media items uploaded to Snapchat's platform) as a surveillance tool to monitor the social unrest in Minneapolis - Saint Paul following the death of George Floyd, making specific use of the manner in which Snapchat presents Snaps on a publicly accessible map. We demonstrate how our technique can be used as a distributed surveillance system supplementing traditional CCTV footage where none would be available. We note a heavy reliance on trust is implied on the geolocation metadata and the resolution of the uploaded media. As a result, our process is vulnerable to database poisoning attacks. We therefore recommend that alternative forensic methods be used to verify and validate surveillance using our method.

A Distributed Surveillance System With Full Coverage Guarantee Using Positive Orthogonal Codes

Surveillance systems provide continual coverage of target area(s) using several cameras through different angles. Conventionally, a central unit controls the system by adjusting the coverage rate of the cameras. However, in large-scale environments, such a centralized system is costly and energy inefficient, as the central unit should exchange a lot of control messages with the cameras. Centralized systems also are not resilient because they fail when the central unit is faulty. In this paper, we propose a distributed surveillance system in which each camera independently decides its activation sequence without exchanging control messages while providing guaranteed coverage of the target areas and ensuring a fault-tolerant system. Our goal is to provide full coverage of all target areas with reduced number of activations of each camera to reduce energy consumption and increase the system lifetime in case of wireless camera sensor network (WCSN) where cameras are battery-powered. To achieve that goal, each camera is activated according to positive orthogonal codes (POCs). POCs have been used for medium access control (MAC) in vehicular sensor networks as a distributed solution for medium access to reduce packet collisions. To the best of our knowledge, this is the first work that uses POCs to create a distributed surveillance system. This work proposes the system design, modeling, and simulation study of a POC-based distributed surveillance system. Moreover, we present a performance evaluation for the coverage percentage and coverage cost metrics using simulations and theoretical analysis. The results indicate that the use of POCs can achieve full coverage of the target areas with a significantly reduced number of cameras compared to a benchmark scheme that uses synchronous fixed repetition (SFR) codes.

Control-Based Resource Management for Storage of Video Streams

Distributed surveillance systems typically consist of multiple cameras that need to store some fraction of their video streams at a central storage node. The disk space of this node constitutes a shared resource. In the paper the disk space allocation is formulated as a PI control problem and a new method for enforcing global resource constraints inspired by anti-windup tracking is proposed. The approach is evaluated by simulations.

Low-Cost Image Compressive Sensing with Multiple Measurement Rates for Object Detection.

When measurement rates grow, most Compressive Sensing (CS) methods suffer from an increase in overheads of transmission and storage of CS measurements, while reconstruction quality degrades appreciably when measurement rates reduce. To solve these problems in real scenarios such as large-scale distributed surveillance systems, we propose a low-cost image CS approach called MRCS for object detection. It predicts key objects using the proposed MYOLO3 detector, and then samples the regions of the key objects as well as other regions using multiple measurement rates to reduce the size of sampled CS measurements. It also stores and transmits half-precision CS measurements to further reduce the required transmission bandwidth and storage space. Comprehensive evaluations demonstrate that MYOLO3 is a smaller and improved object detector for resource-limited hardware devices such as surveillance cameras and aerial drones. They also suggest that MRCS significantly reduces the required transmission bandwidth and storage space by declining the size of CS measurements, e.g., mean Compression Ratios (mCR) achieves 1.43–22.92 on the VOC-pbc dataset. Notably, MRCS further reduces the size of CS measurements by half-precision representations. Subsequently, the required transmission bandwidth and storage space are reduced by one half as compared to the counterparts represented with single-precision floats. Moreover, it also substantially enhances the usability of object detection on reconstructed images with half-precision CS measurements and multiple measurement rates as compared to its counterpart, using a single low measurement rate.

Understanding temporal structure for video captioning

Recent research in convolutional and recurrent neural networks has fueled incredible advances in video understanding. We propose a video captioning framework that achieves the performance and quality necessary to be deployed in distributed surveillance systems. Our method combines an efficient hierarchical architecture with novel attention mechanisms at both the local and global levels. By shifting focus to different spatiotemporal locations, attention mechanisms correlate sequential outputs with activation maps, offering a clever way to adaptively combine multiple frames and locations of video. As soft attention mixing weights are solved via back-propagation, the number of weights or input frames needs to be known in advance. To remove this restriction, our video understanding framework combines continuous attention mechanisms over a family of Gaussian distributions. Our efficient multistream hierarchical model combines a recurrent architecture with a soft hierarchy layer using both equally spaced and dynamically localized boundary cuts. As opposed to costly volumetric attention approaches, we use video attributes to steer temporal attention. Our fully learnable end-to-end approach helps predict salient temporal regions of action/objects in the video. We demonstrate state-of-the-art captioning results on the popular MSVD, MSR-VTT and M-VAD video datasets and compare several variants of the algorithm suitable for real-time applications. By adjusting the frame rate, we show a single computer can generate effective video captions for 100 simultaneous cameras. We additionally perform studies to show how bit rate compression modifies captioning results.

Battery-Saving Communication Modes for Wireless Freeway Traffic Sensors

Efficient management of a freeway network requires continuous decision making on the basis of measured traffic conditions. These measurements usually come from fixed-point sensors deployed in a manner that requires communication links that are always on and are polled at regular intervals. Most of the time, when the sensor data are received at a traffic management center (TMC), no action is taken in response to the information. The high power consumption for the frequent transmissions becomes burdensome for wireless sensors that rely on batteries to last through periods without sufficient illumination for solar power or for the entire life span of the sensor if it has no external power supply. Radio transmissions are a large power draw, so each transmission that can be avoided directly translates into longer battery life. To reduce communication frequency without a significant loss in the quality of information, a distributed freeway surveillance system was developed that prefilters the data at the sensor unit. Five communication modes were developed that assess the value of the measurements before the decision is made to send or not send the measurements. Key to this event-driven approach is that the receiving end is an intelligent part of the distributed surveillance system (i.e., given the lack of transmission, the TMC will know how to interpret the evolving traffic state at the sensor location in the context of previously received information).

Distributed System for Crossroads Traffic Surveillance with Prediction of Incidents

The development of traffic surveillance systems is one of the crucial tasks in intelligent urban surveillance. The visual tracking techniques become more complex with a high computational cost. At the same time, they provide the wide possibilities for motion estimation and prediction in cluttered video sequences. Our contribution is a reasonable application of fast motion estimation with additional using of the clustering procedure. Then the Kalman filter is applied for vehicles’ motion analysis, and the particle filter is used for analysis of pedestrians’ behavior assuming that pedestrians are the weakly predictable objects on the crossroads. Also the distributed surveillance system based on the cloud and fog technologies was designed to process large volumes of video information provided from several IP-cameras in a real-time mode, when six full frames per s are transmitted.

Data modeling and query processing for distributed surveillance systems

This paper presents data modeling and query processing for distributed surveillance systems. We define a metadata rule to search and manage information for distributed or heterogeneous surveillance systems. For human activity recognition, we propose a method that classifies these actions separately from complicated activities as a sequence of basic activities. In addition, we define the domain and range of relations based on the relationship between elements. Furthermore, we describe the state descriptors to represent an image sequence. To address the interaction of multiple objects, we classify human actions into symmetric or asymmetric actions. The prior motion model and the inference approach are applied adaptively according to environments. We define the grammar for the representation of the surveillance video and specify different query criteria for surveillance video retrieval. In the experiments, we show the prototype system that provides event detection, object identification, object tracking, face recognition, and activity recognition.

Business processes in a distributed surveillance system integrated through workflow

SIMPOZ project aims at building a highly configurable system for surveillance of public areas and objects of a special importance based on the analysis of digital images. Distributed nature and potential diversity of monitoring features, a large number of alarm signals fed by various sensors, the need to ensure a flexible event flow management and tailored to needs emergency response processes were motivations to utilize workflow as a management and integration layer. Implementation of workflow processes was preceded by business modeling with ArchiMate language. The presented models of business processes reflect the specificity and characteristics of surveillance systems: reactive character, event driven control and a large number of asynchronous data flows. Lessons learned indicate, that ArchiMate models in spite of the smaller set of dedicated behavioral constructs with respect to for example BPMN, after adopting certain modeling conventions, allow to reach high level of expressiveness.

System-level approach to the design of a smart distributed surveillance system using systemj

Distributed surveillance systems represent a class of sensor networks used for object location and tracking, road traffic monitoring, security, and other purposes. They are very complex to describe, design, and run. Because of their sensitivity, they need to be carefully designed and validated. We present a system-level approach to modeling and designing such systems using a new system-level programming language, SystemJ, which enables designers to describe computational and communication parts of such applications in a highly abstract manner. The designed system can be modeled and validated even before deployment and in that way contribute to the overall reliability and trustworthiness of such systems. As an additional tool, the design environment for specification of the surveillance system topology, physical and communication properties, selected sensors and their interconnectivity with the computing resources was developed. This tool enables easy composition of multiple sensors and their respective controllers, capturing changes of configuration of the system and underlying communication, and automatic generation of the formal description of the surveillance system. This description is then used for the generation of executable code and/or the templates for detailed SystemJ application-specific code, as well as for generation of the operator GUI in a surveillance system.

Self-localizing smart camera networks

This article describes a novel approach to localizing networks of embedded cameras and sensors. In this scheme, the cameras and the sensors are equipped with controllable light sources (either visible or infrared), which are used for signaling. Each camera node can then determine automatically the bearing to all of the nodes that are visible from its vantage point. By fusing these measurements with the measurements obtained from onboard accelerometers, the camera nodes are able to determine the relative positions and orientations of other nodes in the network. The method uses angular measurements derived from images, rather than range measurements derived from time-of-flight or signal attenuation. The scheme can be implemented relatively easily with commonly available components, and it scales well since the localization calculations exploit the sparse structure of the system of measurements. Additionally, the method provides estimates of camera orientation which cannot be determined solely from range measurements. The localization technology could serve as a basic capability on which higher-level applications could be built. The method could also be used to automatically survey the locations of sensors of interest, to implement distributed surveillance systems, or to analyze the structure of a scene, based on images obtained from multiple registered vantage points. It also provides a mechanism for integrating the imagery obtained from the cameras with the measurements obtained from distributed sensors.

Video Sensor Architecture for Surveillance Applications

This paper introduces a flexible hardware and software architecture for a smart video sensor. This sensor has been applied in a video surveillance application where some of these video sensors are deployed, constituting the sensory nodes of a distributed surveillance system. In this system, a video sensor node processes images locally in order to extract objects of interest, and classify them. The sensor node reports the processing results to other nodes in the cloud (a user or higher level software) in the form of an XML description. The hardware architecture of each sensor node has been developed using two DSP processors and an FPGA that controls, in a flexible way, the interconnection among processors and the image data flow. The developed node software is based on pluggable components and runs on a provided execution run-time. Some basic and application-specific software components have been developed, in particular: acquisition, segmentation, labeling, tracking, classification and feature extraction. Preliminary results demonstrate that the system can achieve up to 7.5 frames per second in the worst case, and the true positive rates in the classification of objects are better than 80%.

Intelligent Distributed Surveillance System for People Reidentification in a Transportation Environment

This article presents a solution to track people across a network of cameras with disjoint fields of vision. First, an appearance-based signature is extracted from each frame of the sequence characterizing the passage of a person. This feature, called color-position signature, merges spatial and color information for improved robustness. Moreover, an illuminant invariant procedure has been introduced to manage lighting and camera response changes. Second, the distance between 2 sequences is estimated with dimensionality reduction techniques. Two methods are implemented and compared. The first one is the classical, linear, principal components analysis, whereas the second is a recent nonlinear method called the Laplacian eigenmaps approach. Results from 2 databases acquired in difficult conditions show that the Laplacian eigenmaps method always yields the best results, which confirms the interest of nonlinear approaches in such applications. Moreover, these results show the relevance of this approach combining appearance-based signature and dimensionality reduction technique within the scope of people tracking across distinct fields of vision.

Cooperation in distributed surveillance systems for dense regions

In this paper, we envision a distributed surveillance system where spatially distributed sensors of different modalities can sense collaboratively and continuously large-scale, dynamic, dense and semi-structured environments. The paper discusses surveillance operations in complex and dense environments, such as littoral regions, and describes the key features of distributed surveillance systems. Effective and robust cooperation among the artificial and/or the human agents can synergistically improve the performance of these systems and can endow them with higher-level faculties, such as cooperative target detection and tracking. Different forms of cooperative activities in distributed surveillance systems are mentioned. The paper focuses on intra and inter-platform target cueing and handoff as augmentative forms of cooperation in distributed surveillance.

De-ghosting of tomographic images in a radar network with sparse angular sampling

Taking into account sparsity of the reflectivity function of several radar targets of interest, efficient low-complexity automatic target recognition (ATR) systems can be designed. Such ATR systems would be based on two-dimensional (2D) spatial target models of low dimensionality, where critical information on the radar target signature is summarized. Discrete 2D radar target models can be estimated using high range resolution (HRR) data, measured at a sparse system of view angles. This being the main objective, incoherent tomographic processing of HRR data from a distributed surveillance system, made up of several radar nodes, is studied in this paper. A sparse angular sampling scheme is proposed, which exploits diversity due to both the distributed radar system and the target motion. The novelty is in the exploitation of this locally dense, but otherwise sparse set of viewing angles of the targets, obtained using a sparse network of radars. The de-ghosting efficiency of such a sampling scheme is demonstrated geometrically. This results in identification of minimal information resources for unambiguous estimation of a 2D target model, useful for radar target classification.