Multi-camera Surveillance System Research Articles

Personalized Federated Mutual Learning for Unsupervised Camera-aware Person Re-identification

Person re-identification (ReID) is essential for enhancing security and tracking in multi-camera surveillance systems. To achieve effective person re-identification (ReID) performance across diverse datasets, the Federated Unsupervised Person Re-identification via Camera-aware Clustering (FedUCA) approach has made strides in utilizing distributed datasets while ensuring data privacy. Nevertheless, its uniform model may not adequately cater to the specific characteristics of each participant’s data, given the diversity in camera perspectives and client-specific data variances, thus obtaining degraded results. To address this issue, we propose an advanced framework, Personalized Federated Dual-Model Learning for Camera-Aware Person Re-Identification (PerFedDual), which introduces knowledge-sharing techniques inherent to mutual learning for FedUCA with the camera-centric clustering process. PerFedDual supports a dual-model training approach that creates a cooperative learning space that improves both the global model and client-specific models by exchanging knowledge both ways. The methodology adopted is precisely adjusted to the distinctive data environment of each client, ensuring the protection of privacy while simultaneously enhancing the accuracy and flexibility of ReID models in diverse camera configurations. The empirical evaluation reveals that PerFedDual outperforms FedUCA and alternative federated learning strategies, highlighting the benefits of our technique that leverage collective intelligence to enhance unsupervised person re-identification.

Temporal Correlation Vision Transformer for Video Person Re-Identification

Video Person Re-Identification (Re-ID) is a task of retrieving persons from multi-camera surveillance systems. Despite the progress made in leveraging spatio-temporal information in videos, occlusion in dense crowds still hinders further progress. To address this issue, we propose a Temporal Correlation Vision Transformer (TCViT) for video person Re-ID. TCViT consists of a Temporal Correlation Attention (TCA) module and a Learnable Temporal Aggregation (LTA) module. The TCA module is designed to reduce the impact of non-target persons by relative state, while the LTA module is used to aggregate frame-level features based on their completeness. Specifically, TCA is a parameter-free module that first aligns frame-level features to restore semantic coherence in videos and then enhances the features of the target person according to temporal correlation. Additionally, unlike previous methods that treat each frame equally with a pooling layer, LTA introduces a lightweight learnable module to weigh and aggregate frame-level features under the guidance of a classification score. Extensive experiments on four prevalent benchmarks demonstrate that our method achieves state-of-the-art performance in video Re-ID.

Improving consistency of AADL models: A composition approach

AbstractAADL—architecture analysis and design language—has proven to be important in developing real‐time embedded systems. However, its formalization capability is still limited, which makes it difficult to produce a consistent model; particularly, in the case where several models are combined together. Therefore, using formal verification is an effective way to meet the consistency requirements of the AADL model. This paper proposes a model‐based engineering approach that uses timed automata (TA) formalism to aggregate multiple AADL models into a single composite model. Based on the set of input AADL models, the proposed approach creates a consistent composed model as well as support model checking verification. The effectiveness of this contribution is evaluated through two case studies; a multicamera surveillance system and a drone system.

Pose-driven attention-guided image generation for person re-Identification

Person re-identification (re-ID) concerns the matching of subject images across different camera views in a multi camera surveillance system. One of the major challenges in person re-ID is pose variations across the camera network, which significantly affects the appearance of a person. Existing development data lack adequate pose variations to carry out effective training of person re-ID systems. To solve this issue, in this paper we propose an end-to-end pose-driven attention-guided generative adversarial network, to generate multiple poses of a person. We propose to attentively learn and transfer the subject pose through an attention mechanism. A semantic-consistency loss is proposed to preserve the semantic information of the person during pose transfer. To ensure fine image details are realistic after pose translation, an appearance discriminator is used while a pose discriminator is used to ensure the pose of the transferred images will exactly be the same as the target pose. We show that by incorporating the proposed approach in a person re-identification framework, realistic pose transferred images and state-of-the-art re-identification results can be achieved.

Adaptive Method for Modeling of Temporal Dependencies between Fields of Vision in Multi-Camera Surveillance Systems

A method of modeling the time of object transition between given pairs of cameras based on the Gaussian Mixture Model (GMM) is proposed in this article. Temporal dependencies modeling is a part of object re-identification based on the multi-camera experimental framework. The previously utilized Expectation-Maximization (EM) approach, requiring setting the number of mixtures arbitrarily as an input parameter, was extended with the algorithm that automatically adapts the model to statistical data. The probabilistic model was obtained by matching to the histogram of transition times between a particular pair of cameras. The proposed matching procedure uses a modified particle swarm optimization (mPSO). A way of using models of transition time in object re-identification is also presented. Experiments with the proposed method of modeling the transition time were carried out, and a comparison between previous and novel approach results are also presented, revealing that added swarms approximate normalized histograms very effectively. Moreover, the proposed swarm-based algorithm allows for modelling the same statistical data with a lower number of summands in GMM.

A Review On State Of The Art Abnormal Activity Recognition Approaches

In last few decades, technological revolution has accelerated the deployment of large scale surveillance systems on almost all public places such as malls, hospitals, airports, railways, bus stations, roads, etc. These intelligent surveillance systems can play crucial role in governance of situations, collective security and safety, mitigating as well as prevention of adversaries. With gradual increase in multi camera surveillance systems enclosing multi angle views of same as well as different scenes has increased complexity of monitoring the systems by manual inspection. Abnormalities also known as anomalies or outliers are inevitable part of the existence and presumed to be rare in occurrence. Manual monitoring of such abnormalities is susceptible to errors and limited by human capabilities such as inattention and tiresome. Hence in the field of computer vision, automated abnormal activity recognition (AAR) from surveillance systems is emerging research area. The intent of this research is to shed a light on recent innovations and developments that have made a mark in abnormal activity recognition (AAR) involving deep learning. This paper also includes conventional categorization of anomalies based on different perspectives which can provide better understanding to young researchers. Though recent developments still poses many real time challenges in automatic abnormal activity recognition, some of them are enclosed in this paper.

Person Retrieval in Video Surveillance Using Deep Learning–Based Instance Segmentation

Video surveillance systems are deployed at many places such as airports, train stations, and malls for security and monitoring purposes. However, it is laborious to search for and retrieve persons in multicamera surveillance systems, especially with cluttered backgrounds and appearance variations among multiple cameras. To solve these problems, this paper proposes a person retrieval method that extracts the attributes of a masked image using an instance segmentation module for each object of interest. It uses attributes such as color and type of clothes to describe a person. The proposed person retrieval system involves four steps: (1) using the YOLACT++ model to perform pixelwise person segmentation, (2) conducting appearance‐based attribute feature extraction using a multiple convolutional neural network classifier, (3) employing a search engine with a fundamental attribute matching approach, and (4) implementing a video summarization technique to produce a temporal abstraction of retrieved objects. Experimental results show that the proposed retrieval system can achieve effective retrieval performance and provide a quick overview of retrieved content for multicamera surveillance systems.

Review of GAN-Based Person Re-Identification

Person re-ID is becoming increasingly popular in the field of modern surveillance. The purpose of person re-ID is to retrieve person of interests in non-overlapping multi-camera surveillance system. Due to the complexity of the surveillance scene, the person images captured by cameras often have problems such as size variation, rotation, occlusion, illumination difference, etc., which brings great challenges to the study of person re-ID. In recent years, studies based on deep learning have achieved great success in person re-ID. The improvement of basic networks and a large number of studies on the influencing factors have greatly improved the accuracy of person re-ID. Recently, some studies utilize GAN to tackle the domain adaptation task by transferring person images of source domain to the style of target domain and have achieved state of the art result in person re-ID.

Tensor-based sparse canonical correlation analysis via low rank matrix approximation for RGB-D long-term person re-identification

Person re-identification can be a part of almost any multi-camera surveillance systems. Most previous works propose strategies for short-term person re-identification which are usually driven from appearance features of RGB images. However, when people appear in excessive lighting or change clothes (i.e. long-term case), short-term person re-identification approaches have a tendency to fail. In this paper, we propose a novel approach for long-term person re-identification by employing depth videos of RGB-D sensors. We also develop a sparse canonical correlation analysis using a local third-order tensor model to accomplish multi-level person re-identification. The tensor representations of images make the space for performing the multi-level person re-identification simpler compared to existing methods. Finally, we evaluate our experiments on RGB-D long-term datasets consisting of BIWI RGBD-ID dataset and IAS-Lab RGBD-ID dataset. The results demonstrate the efficiency of the proposed method compared to recent methods.

Vision-based nonintrusive context documentation for earthmoving productivity simulation

Abstract Although video surveillance systems have shown potential for analyzing jobsite contexts, the necessity of a complex multi-camera surveillance system or workers' privacy issues remain as substantive hurdles to adopt such systems in practice. To address such issues, this study presents a non-intrusive earthmoving productivity analysis method using imaging and simulation. The site access log of dump trucks is used to infer earthmoving contexts, which is produced by analyzing videos recorded at the entrance and the exit of a construction site. An algorithm for license plate detection and recognition in an uncontrolled environment is developed to automatically produce the site access log, by leveraging video deinterlacing, a deep convolutional network, and rule-based post-processing. The experimental results show the effectiveness of the proposed method for producing the site access log. Based on the site access log, simulation-based productivity analysis is conducted to produce a daily productivity report, which can provide the basis for earthmoving resource planning. It is expected that the resulting daily productivity report promotes data-driven decision-making for earthmoving resource allocation, thereby improving potential for saving cost and time for earthworks with an updated resource allocation plan.

Random Occlusion Recovery for Person Re-identification

As a basic task of multi-camera surveillance system, person re-identification aims to re-identify a query pedestrian observed from non-overlapping multiple cameras or across different time with a single camera. Recently, deep learning-based person re-identification models have achieved great success in many benchmarks. However, these supervised models require a large amount of labeled image data, and the process of manual labeling spends much manpower and time. In this study, we introduce a method to automatically synthesize labeled person images and adopt them to increase the sample number per identity for person re-identification datasets. To be specific, we use block rectangles to randomly occlude pedestrian images. Then, a generative adversarial network (GAN) model is proposed to use paired occluded and original images to synthesize the de-occluded images that similar but not identical to the original image. Afterwards, we annotate the de-occluded images with the same labels of their corresponding raw images and use them to augment the number of samples per identity. Finally, we use the augmented datasets to train baseline model. The experiment results on CUHK03, Market-1501 and DukeMTMC-reID datasets show that the effectiveness of the proposed method.

Surveillance-Aware Uplink Scheduling for Cellular Networks

Most scheduling algorithms in the literature for cellular networks are concerned with throughput, fairness, or cost optimization. Recently, however, wireless surveillance in cellular networks has become increasingly important, and more and more institutions and companies have adopted commercial cellular surveillance cameras due to their low installation cost and the wide network coverage. In this paper, therefore, we first explore the resource allocation problem for a multi-camera surveillance system in cellular networks. We minimize the number of allocated resource blocks (RBs) while simultaneously ensuring the coverage requirement for the surveillance system in cellular networks. Specifically, we first describe our system model and then formulate the Camera Set Resource Allocation Problem (CSRAP). Next, we prove that the problem is NP-hard and inapproximable within ln n, where n is the number of surveillance targets. To solve the problem, we propose an approximation algorithm for the general case of CSRAP and then we find the optimal solutions of three deployments of cameras in the Manhattan Street Network to find the intrinsic characteristics of camera selections. The simulation results, based on two real surveillance maps and synthetic datasets, show that the number of allocated RBs can be effectively reduced compared to the existing approach for cellular networks.

Histogram of the node strength and histogram of the edge weight: two new features for RGB-D person re-identification

Person re-identification is a classical task for any multi-camera surveillance system. Most of the existing researches on re-identification are based on features extracted from RGB images. However, there are many deficiencies in RGB image processing, some of which are requiring a lot of illumination and high computation. In this paper, novel features are proposed for RGB-D person re-identification. First, the complex network approach in texture recognition is modified and its threshold function is changed for using in depth images extracted by RGB-D sensors. Then, two novel measurements named the histogram of the edge weight (HEW) and the histogram of the node strength (HNS) are introduced on complex networks. Our features fit both single-shot and multi-shot person re-identification. In the single-shot case, the HNS is extracted from only one frame while for the multi-shot case it is extracted from both one frame and multi-frames. These proposed measurements are called histogram of the spatial node strength (HSNS) and histogram of the temporal node strength (HTNS) respectively. Subsequently, these measurements are combined with skeleton features using score-level fusion. The method is evaluated using two benchmark databases and the results show that ours outperforms some state-of-the-art methods.

Person Reidentification Using Multiple Egocentric Views

Development of a robust and scalable multicamera surveillance system is the need of the hour to ensure public safety and security. Being able to reidentify and track one or more targets over multiple nonoverlapping camera field of views in a crowded environment remains an important and challenging problem because of occlusions, large change in the viewpoints, and illumination across cameras. However, the rise of wearable imaging devices has led to new avenues in solving the reidentification (re-id) problem. Unlike static cameras, where the views are often restricted or low resolution and occlusions are common scenarios, egocentric/first person views (FPVs) mostly get zoomed in, unoccluded face images. In this paper, we present a person re-id framework designed for a network of multiple wearable devices. The proposed framework builds on commonly used facial feature extraction and similarity computation methods between camera pairs and utilizes a data association method to yield globally optimal and consistent re-id results with much improved accuracy. Moreover, to ensure its utility in practical applications where a large amount of observations are available every instant, an online scheme is proposed as a direct extension of the batch method. This can dynamically associate new observations to already observed and labeled targets in an iterative fashion. We tested both the offline and online methods on realistic FPV video databases, collected using multiple wearable cameras in a complex office environment and observed large improvements in performance when compared with the state of the arts.

Multi-camera surveillance systems for time and motion studies of timber harvesting equipment

We evaluated the feasibility of using a multi-camera security system to conduct time and motion studies. It was installed on a John Deere 540G cable skidder and connected to the skidder’s battery for continuous recording with minimal effort and intervention. After recording the skidder’s work for eleven experimental skidding cycles, time stamped video footage was visually inspected to obtain time consumption of work tasks, which provided for accurate calculation of total cycle times and delays. Several advantages of the security camera system including quick and non-invasive installation, large memory storage, transferability, resistance to weather elements, and the capacity to capture different views, offer a great potential for this method to be adopted as a reliable approach to accurately conduct time and motion studies. Along with distance and gradient information for skid-trail segments, we also explored the influence of gradient on travel time for loaded and unloaded skidding. There is a need for future studies to formally explore this relationship and develop more detailed cycle time equations that explicitly take into account skid-trail gradient for individual segments.

Fully-automated person re-identification in multi-camera surveillance system with a robust kernel descriptor and effective shadow removal method

In this paper, a fully-automated person Re-ID (Re-identification) system is proposed for real scenarios of human tracking in non-overlapping camera network. The system includes two phases of human detection and Re-ID. The human ROIs (Regions of Interest) are extracted from human detection phase and then feature extraction is done on these ROIs in order to build human descriptor for Re-ID. Unlike other approaches which deal with manually-cropped human ROIs for person Re-ID, in this system, the person identity is determined based on the human ROIs extracted automatically by a combined method of human detection. Two main contributions are proposed on both phases of human detection and Re-ID in order to enhance the performance of person Re-ID system. First, an effective shadow removal method based on score fusion of density matching is proposed to get better human detection results. Second, a robust KDES (Kernel DEScriptor) is extracted from human ROI for person classification. Additionally, a new person Re-ID dataset is built in real surveillance scenarios from multiple cameras. The experiments on benchmark datasets and our own dataset show that the person Re-ID results using the proposed solutions outperform some of the state-of-the-art methods.

Improved multi-camera smart traffic surveillance system for resilient cities

In this paper, intelligent traffic surveillance system as an important part of a smart resilient city is reviewed. The smart traffic control system is very important in improving the life style by decreasing the traffic saturation and air pollution. Still, multi-camera vision is more helpful in implementing different automatic traffic surveillance systems. Apart from their superior features, existing multi-camera surveillance systems suffer from computational complexity and degraded accuracy. The main reason for these shortcomings arises from image processing errors. These errors depend on the image content and thus are not predictable. To overcome this shortage, three-dimensional (3D) optical techniques for improved fusing the multi-camera images and thus extracting 3D vehicle locations is proposed in this paper. In fact the proposed multi-camera visionary system is a combination of image processing based and geometrical optics based methods. The result of 3D image reconstruction through the proposed technique shows its dominance in providing the 3D image information.

Time-based feedback-control framework for real-time video surveillance systems with utilization control

Multi-camera surveillance systems generally are used by intelligent vision applications to analysis moving objects in the urban environments. Object tracking is an essential component of these applications. Real-time tracking of moving objects in the crowded urban scenes is a challenging problem because the number of moving objects in the scene varies and is usually unpredictable. In order to meet the real-time requirements of intelligent vision applications, it is necessary to control the surveillance workload over processing system and enforce the utilization bound on the system. The utilization control is challenging especially when the workload in the system is unpredictable. The workload for a particular camera depending on the number of targets in its view and execution rate of services that are used to detect and track the moving objects. In this paper, an adaptive real-time system based on the feedback-control framework is proposed to control the system utilization by adjusting input video frame rate that guarantees real-time performance and quality of intelligent vision applications. This approach can handle the workload uncertainties by identifying the parameters of system model online by solving an optimization problem and can simultaneously adjust the frame rate in each sampling period. In each sampling instance, the system utilization is monitored and the error is obtained by comparing it with the set point. The real-time operation of the computing system is achieved by adjusting the input video frame rate to keep the utilization at a given set point slightly below their schedulable bound. Evaluation results demonstrate proposed scheme outperforms the existing schemes especially under workload uncertainties.

Deep Ranking for Person Re-Identification via Joint Representation Learning.

This paper proposes a novel approach to person re-identification, a fundamental task in distributed multi-camera surveillance systems. Although a variety of powerful algorithms have been presented in the past few years, most of them usually focus on designing hand-crafted features and learning metrics either individually or sequentially. Different from previous works, we formulate a unified deep ranking framework that jointly tackles both of these key components to maximize their strengths. We start from the principle that the correct match of the probe image should be positioned in the top rank within the whole gallery set. An effective learning-to-rank algorithm is proposed to minimize the cost corresponding to the ranking disorders of the gallery. The ranking model is solved with a deep convolutional neural network (CNN) that builds the relation between input image pairs and their similarity scores through joint representation learning directly from raw image pixels. The proposed framework allows us to get rid of feature engineering and does not rely on any assumption. An extensive comparative evaluation is given, demonstrating that our approach significantly outperforms all the state-of-the-art approaches, including both traditional and CNN-based methods on the challenging VIPeR, CUHK-01, and CAVIAR4REID datasets. In addition, our approach has better ability to generalize across datasets without fine-tuning.

A framework for semantic people description in multi-camera surveillance systems

People re-identification has been a very active research topic recently in computer vision. It is an important application in surveillance systems with disjoint cameras. In this paper, a framework is proposed to extract descriptors of people in videos, which are based on soft-biometric traits and can be further used for people re-identification or other applications. Soft-biometric based description is more invariant to changing factors than directly using low level features such as color and texture. The ensemble of a set of soft-biometric traits can achieve good performance in people re-identification. In the proposed method, the body of detected people is divided into three parts and the selected soft-biometric traits are extracted from each part. All traits are then combined to form the final descriptor, and people re-identification is performed based on the descriptor and Nearest Neighbor (NN) matching strategy. The experiments are carried out on SAIVT-SoftBio database which consists of videos from disjoint surveillance cameras, as well as some static image based datasets. An open ID recognition problem is also evaluated for the proposed method. Comparisons with some state-of-the-art methods are provided as well. The experiment results show the good performance of the proposed framework.