Multi-camera Video Surveillance Research Articles

Cross-Camera Tracking Model and Method Based on Multi-Feature Fusion

Multi-camera video surveillance has been widely applied in crowd statistics and analysis in smart city scenarios. Most existing studies rely on appearance or motion features for cross-camera trajectory tracking, due to the changing asymmetric perspectives of multiple cameras and occlusions in crowded scenes, resulting in low accuracy and poor tracking performance. This paper proposes a tracking method that fuses appearance and motion features. An implicit social model is used to obtain motion features containing spatio-temporal information and social relations for trajectory prediction. The TransReID model is used to obtain appearance features for re-identification. Fused features are derived by integrating appearance features, spatio-temporal information and social relations. Based on the fused features, multi-round clustering is adopted to associate cross-camera objects. Exclusively employing robust pedestrian reidentification and trajectory prediction models, coupled with the real-time detector YOLOX, without any reliance on supplementary information, an IDF1 score of 70.64% is attained on typical datasets derived from AiCity2023.

Improved Baited Remote Underwater Video (BRUV) for 24 h Real-Time Monitoring of Pelagic and Demersal Marine Species from the Epipelagic Zone

Bait-based remote underwater video (BRUV) systems are effective devices for remotely observing fish and other marine organisms in challenging environments. The development of a long duration (24 h) surface BRUV observation surveys allowed the monitoring of scarce and elusive pelagic sharks and the direct impact on non-targeted species of longline fishing in the Western Mediterranean. Technological limitations, such as the limited storage capacity and a single surface camera, were improved by (i) adding a deep camera equipped with light (below 80 m depth) and (ii) replacing Gopros with a multi-camera video surveillance system (surface and depth) with a storage capacity of several days and access to real-time observation. Based on a deployment effort of 1884 h video data, we identified 11 blue sharks (Prionace glauca) and one bluntnose sixgill shark (Hexanchus griseus), a deep-sea species that scarcely swims at the surface. The real-time observation capability was a powerful tool for reducing logistical costs and for raising environmental awareness in educational and outreach programmes.

Incremental multi-target domain adaptation for object detection with efficient domain transfer

Recent advances in unsupervised domain adaptation have significantly improved the recognition accuracy of CNNs by alleviating the domain shift between (labeled) source and (unlabeled) target data distributions. While the problem of single-target domain adaptation (STDA) for object detection has recently received much attention, multi-target domain adaptation (MTDA) remains largely unexplored, despite its practical relevance in several real-world applications, such as multi-camera video surveillance. Compared to the STDA problem that may involve large domain shifts between complex source and target distributions, MTDA faces additional challenges, most notably the computational requirements and catastrophic forgetting of previously-learned targets, which can depend on the order of target adaptations. STDA for detection can be applied to MTDA by adapting one model per target, or one common model with a mixture of data from target domains. However, these approaches are either costly or inaccurate. The only state-of-art MTDA method specialized for detection learns targets incrementally, one target at a time, and mitigates the loss of knowledge by using a duplicated detection model for knowledge distillation, which is computationally expensive and does not scale well to many domains. In this paper, we introduce an efficient approach for incremental learning that generalizes well to multiple target domains. Our MTDA approach is more suitable for real-world applications since it allows updating the detection model incrementally, without storing data from previous-learned target domains, nor retraining when a new target domain becomes available. Our approach leverages domain discriminators to train a novel Domain Transfer Module (DTM), which only incurs a modest overhead. The DTM transforms source images according to diverse target domains, allowing the model to access a joint representation of previously-learned target domains, and to effectively limit catastrophic forgetting. Our proposed method – called MTDA with DTM (MTDA-DTM) – is compared against state-of-the-art approaches on several MTDA detection benchmarks and Wildtrack, a benchmark for multi-camera pedestrian detection. Results indicate that MTDA-DTM achieves the highest level of detection accuracy across multiple target domains, yet requires significantly fewer computational resources. Our code is available.11https://github.com/Natlem/M-HTCN.

Spatio-Temporal association rule based deep annotation-free clustering (STAR-DAC) for unsupervised person re-identification

Multi-camera video surveillance environment has a variety of emerging research problems among, which person re-identification is the premier one. Unsupervised person re-identification has been explored less in literature than the supervised approach. Images acquired from the video surveillance systems are unlabeled, which denotes that it is naturally an unsupervised learning problem. The state-of-the-art unsupervised methods seek external annotations support such as incorporating transfer learning techniques, partial labeling of train images, etc., which makes them not purely unsupervised and unsuitable for practical real-world surveillance settings. Identity mismatch happens due to the similar costumes and complex environmental factors. To resolve this issue, we introduce a new framework named Spatio-Temporal Association Rule based Deep Annotation-free Clustering (STAR-DAC) which incrementally clusters the unlabeled person re-identification images based on visual features and performs cluster fine-tuning through the mined spatio-temporal association rules. STAR formulations leveraged upto 75% of images for reliable sample selection through cluster fine-tuning. STAR based fine-tune algorithm aims to attain ground-truth labels of an unlabeled dataset and eliminate cluster outliers to stabilize the evaluation. Experiments are performed on image and video-based benchmark person re-identification datasets such as DukeMTMC re-ID, Market1501, MSMT17, CUHK03, GRID and Dukevideo re-ID, iLIDSVid, ViPer respectively. Experimental results clearly show that the proposed STAR-DAC framework outperforms the state-of-the-art methods in case of large scale datasets with multiple cameras.

Full-scaled deep metric learning for pedestrian re-identification

The pedestrian re-identification problem (i.e., re-id) is essential and pre-requisite in multi-camera video surveillance studies, provided the fact that pedestrian targets need to be accurately re-identified across a network of multiple cameras with non-overlapping fields of views before other post-hoc high-level utilizations (i.e., tracking, behaviors analyses, activities monitoring, etc.) can be carried out. Driven by recent developments in deep learning techniques, the important re-id problem is often tackled via either deep discriminant learning or deep generative learning techniques. However, most contemporary deep learning-based models with tremendously deep structures are not easy to be trained because of the notorious vanishings gradient problem. In this study, a novel full-scaled deep discriminant learning model is proposed. The novelty of the full-scale model is significant, as three crucial concepts in designing a deep learning model, including depth, width, and cardinality, are all taken into consideration, simultaneously. Therefore, the new model needs not to be tremendously deep but is more convenient to be trained. Moreover, based on the new model, a novel deep metric learning method is proposed to further solve the important re-id problem. Technically, two algorithms either based on the conventional SGD (stochastic gradient descent) or an alternative more efficient PGD (proximal gradient descent) are both derived. For experimental analyses, the newly introduced full-scaled deep metric learning method has been comprehensively compared with dozens of popular re-id methods proposed from either deep learning or shallow learning perspectives. Several well-known public re-id datasets have been incorporated and rigorous statistical analyses have been carried out to compare all methods regarding their re-id performance. The superiority of the novel full-scaled deep metric learning method has been substantiated, from the statistical point of view.

Person Tracking and Reidentification for Multicamera Indoor Video Surveillance Systems

For practical use, the relevance of indoor surveillance from multiple cameras to track the movement of people and reidentify them in video sequences is constantly increasing. This is a complex task due to the effect of uneven illumination, background inhomogeneity, overlap, uncertainty of the trajectories of people, and the similarity of their visual features. The article presents an approach to track people by video sequences and reidentify them in multicamera video surveillance systems that are used indoors. At the first step, people are detected using a YOLO v4 convolution neural network (CNN) and described by a rectangular area. Further, the search for the face area and the calculation of its features are carried out, which in the developed method are used when accompanying a person in a video sequence and during his intercamera reidentification. This approach improves the accuracy of tracking with a complex movement trajectory and multiple intersections of people with similar characteristics. The search for faces is carried out on the detected areas based on the multitasking MTCNN, and the MobileFaceNetwork model is used to form the vector of the features of the face. Human features are generated using a modified CNN based on ResNet34 and an HSV color tone channel histogram. The correspondence between people on different frames is established based on the analysis of the spatial coordinates of faces and people, as well as their CNN features, using the Hungarian algorithm. To ensure the accuracy of intercamera tracking, reidentification is performed based on the facial features. Five test video sequences of different numbers of people captured indoors with a fixed video camera were used to test and compare different approaches. The obtained experimental results confirmed the strength of the characteristics of the proposed approach.

Laplacian LRR on Product Grassmann Manifolds for Human Activity Clustering in Multicamera Video Surveillance

In multicamera video surveillance, it is challenging to represent videos from different cameras properly and fuse them efficiently for specific applications such as human activity recognition and clustering. In this paper, a novel representation for multicamera video data, namely, the product Grassmann manifold (PGM), is proposed to model video sequences as points on the Grassmann manifold and integrate them as a whole in the product manifold form. In addition, with a new geometry metric on the product manifold, the conventional low rank representation (LRR) model is extended onto PGM and the new LRR model can be used for clustering nonlinear data, such as multicamera video data. To evaluate the proposed method, a number of clustering experiments are conducted on several multicamera video data sets of human activity, including the Dongzhimen Transport Hub Crowd action data set, the ACT 42 Human Action data set, and the SKIG action data set. The experiment results show that the proposed method outperforms many state-of-the-art clustering methods.

Guest Editorial Introduction to the Special Issue on Group and Crowd Behavior Analysis for Intelligent Multicamera Video Surveillance

Despite significant progress in human behavior analysis over the past few years, most of today’s state-of-the-art algorithms focus on analyzing individual behavior in a simple environment monitored by a single camera. Recently, the widespread availability of cameras and a growing need for public safety have shifted the attention of researchers in video surveillance from individual behavior analysis to group and crowd behavior analysis in multicamera networks. Group behavior analysis provides a novel level for describing events, which are semantically more meaningful, highlighting barely visible relational connections among people. Crowd behavior analysis can also be used for anomaly detection such as panic scenarios, dangerous situations, and illegal behaviors in public spaces.

Learning from evolving video streams in a multi-camera scenario

Nowadays, video surveillance systems are taking the first steps toward automation, in order to ease the burden on human resources as well as to avoid human error. As the underlying data distribution and the number of concepts change over time, the conventional learning algorithms fail to provide reliable solutions for this setting. In this paper, we formalize a learning concept suitable for multi-camera video surveillance and propose a learning methodology adapted to that new paradigm. The proposed framework resorts to the universal background model to robustly learn individual object models from small samples and to more effectively detect novel classes. The individual models are incrementally updated in an ensemble-based approach, with older models being progressively forgotten. The framework is designed to detect and label new concepts automatically. The system is also designed to exploit active learning strategies, in order to interact wisely with operator, requesting assistance in the most ambiguous to classify observations. The experimental results obtained both on real and synthetic data sets verify the usefulness of the proposed approach.

Special issue on group and crowd behavior analysis for intelligent multi-camera video surveillance

Special issue on group and crowd behavior analysis for intelligent multi-camera video surveillance

움직임 감지를 사용하여 영상 해상도를 자동 제어하는 실시간 다중 카메라 영상 감시 시스템의 구현

본 논문에서는 움직임 감지를 사용하여 영상 해상도를 자동 제어하는 실시간 다중 카메라 영상 감시 시스템을 구현하였다. 평상시에는 4개 채널의 영상을 QVGA급으로 취득한 후 하나의 VGA급 영상으로 통합하여 전송한다. 움직임이 포착되는 경우에는 해당 채널의 영상을 자동으로 확대하여 VGA급으로 취득한 후 나머지 3개 채널의 영상을 QQVGA급으로 줄여서 오버레이한다. 이를 통하여 모든 채널의 영상을 놓치지 않으면서도 전송 대역폭을 늘리지 않고 움직임이 포착된 채널을 확대하여 감시할 수 있다. 0.18 um 공정에서 합성한 최대 동작 주파수는 110 MHz로서 이론상으로 4개의 HD급 카메라를 지원할 수 있다. This paper proposes a real-time multi-camera video surveillance system with automatic resolution control using motion detection. In ordinary times, it acquires 4 channels of QVGA images, and it merges them into single VGA image and transmit it. When motion is detected, it automatically increases the resolution of motion-occurring channel to VGA and decreases those of 3 other channels to QQVGA, and then these images are overlaid and transmitted. Thus, it can magnifies and watches the motion-occurring channel while maintaining transmission bandwidth and monitoring all other channels. When it is synthesized with 0.18 um technology, the maximum operating frequency is 110 MHz, which can theoretically support 4 HD cameras.

A Novel Video Stitching Method for Multi-Camera Surveillance Systems

This paper proposes a novel video stitching method that improves real-time performance and visual quality of a multi-camera video surveillance system. A two-stage seam searching algorithm based on enhanced dynamic programming is proposed. It can obtain satisfactory result and achieve better real-time performance than traditional seam-searching methods. The experiments show that the computing time is reduced by 66.4% using the proposed algorithm compared with enhanced dynamic programming, while the seam-searching accuracy is maintained. A real-time local update scheme reduces the deformation effect caused by moving objects passing through the seam, and a seam-based local color transfer model is constructed and applied to achieve smooth transition in the overlapped area, and overcome the traditional pixel blending methods. The effectiveness of the proposed method is proved in the experiements.

Adaptive Method of Adjusting Flowgraph for Route Reconstruction in Video Surveillance Systems

Pawlak's flowgraph has been applied as a suitable data structure for description and analysis of human behaviour in the area supervised with multicamera video surveillance system. Information contained in the flowgraph can be easily used to predict consecutive movements of a particular object. Moreover, utilization of the flowgraph can support reconstructing object route from the past video images. However, such a flowgraph with its accumulative nature needs a certain period of time for adaptation to changes in behaviour of objects which can be caused, e.g. by closing a door or placing other obstacle forcing people to pass it by. In this paper a method for reduction of time needed for flowgraph adaptation is presented. Additionally, distance measure between flowgraphs is also introduced in order to determine if carrying out the adaptation process is needed.

Intelligent multi-camera video surveillance: A review

Intelligent multi-camera video surveillance is a multidisciplinary field related to computer vision, pattern recognition, signal processing, communication, embedded computing and image sensors. This paper reviews the recent development of relevant technologies from the perspectives of computer vision and pattern recognition. The covered topics include multi-camera calibration, computing the topology of camera networks, multi-camera tracking, object re-identification, multi-camera activity analysis and cooperative video surveillance both with active and static cameras. Detailed descriptions of their technical challenges and comparison of different solutions are provided. It emphasizes the connection and integration of different modules in various environments and application scenarios. According to the most recent works, some problems can be jointly solved in order to improve the efficiency and accuracy. With the fast development of surveillance systems, the scales and complexities of camera networks are increasing and the monitored environments are becoming more and more complicated and crowded. This paper discusses how to face these emerging challenges.

Falling-incident detection and throughput enhancement in a multi-camera video-surveillance system

For most elderly, unpredictable falling incidents may occur at the corner of stairs or a long corridor due to body frailty. If we delay to rescue a falling elder who is likely fainting, more serious consequent injury may occur. Traditional secure or video surveillance systems need caregivers to monitor a centralized screen continuously, or need an elder to wear sensors to detect falling incidents, which explicitly waste much human power or cause inconvenience for elders. In this paper, we propose an automatic falling-detection algorithm and implement this algorithm in a multi-camera video surveillance system. The algorithm uses each camera to fetch the images from the regions required to be monitored. It then uses a falling-pattern recognition algorithm to determine if a falling incident has occurred. If yes, system will send short messages to someone needs to be noticed. The algorithm has been implemented in a DSP-based hardware acceleration board for functionality proof. Simulation results show that the accuracy of falling detection can achieve at least 90% and the throughput of a four-camera surveillance system can be improved by about 2.1 times.

H.264-based Wireless Surveillance Sensors in Application to Target Identification and Tracking

In this paper, the author propose a hardware-based m icro structure inside the architecture of H.264 encoder, called Vector Bank (VB), which can dram atically reduce the bandwidth, mem ory, and com putation tim e of a multi-cam era video surveillance system . VB is a m em ory-based structure that contains the vector inform ation for every single Macro Block (MB) of the video fram e from the Motion Estim ation m odule of the H.264 encoder. After extraction of the several vectors from several fram es, a dedicated Digital Signal Processor (DSP) can be assigned to analyze and predict the trajectories of the objects in m otion. With the application of 2-D edge detectors, such as Sobel or Laplacian of Gaussian (LoG) operators, the objects in m otion can be isolated from the background and thus be easily identified. In addition, a m iddleware-based technique derived from Directional Discrete Cosine Transform (DDCT) is introduced to im prove the im age quality without sacrificing the system perform ance. We propose four new m odes of operations in DDCT for better im age com pression ratio. The experim ental results show that we can im prove the im age quality of the targets, isolate them from the background, and track them easily in both linear and exponential trajectories of the m otion.

Multi-camera video surveillance for real-time analysis and reconstruction of soccer games

Soccer analysis and reconstruction is one of the most interesting challenges for wide-area video surveillance applications. Techniques and system implementation for tracking the ball and players with multiple stationary cameras are discussed. With video data captured from a football stadium, the real-world, real-time positions of the ball and players can be generated. The whole system contains a two-stage workflow, i.e., single view and multi-view processing. The first stage includes categorizing of players and filtering of the ball after changing detection against an adaptive background and image-plane tracking. Occlusion reasoning and tracking-back is applied for robust ball filtering. In the multi-view stage, multiple observations from overlapped single views are fused to refine players’ positions and to estimate 3-D ball positions using geometric constraints. Experimental results on real data from long sequences are demonstrated.

Euclidean path modeling for video surveillance

In this paper, we address the issue of Euclidean path modeling in a single camera for activity monitoring in a multi-camera video surveillance system. The method consists of a path building training phase and a testing phase. During the unsupervised training phase, after auto-calibrating a camera and thereafter metric rectifying the input trajectories, a weighted graph is constructed with trajectories represented by the nodes, and weights determined by a similarity measure. Normalized-cuts are recursively used to partition the graph into prototype paths. Each path, consisting of a partitioned group of trajectories, is represented by a path envelope and an average trajectory. For every prototype path, features such as spatial proximity, motion characteristics, curvature, and absolute world velocity are then recovered directly in the rectified images or by registering to aerial views. During the testing phase, using our simple yet efficient similarity measures for these features, we seek a relation between the trajectories of an incoming sequence and the prototype path models to identify anomalous and unusual behaviors. Real-world pedestrian sequences are used to evaluate the steps, and demonstrate the practicality of the proposed approach.

Distributed Interactive Video Arrays for Event Capture and Enhanced Situational Awareness

Video surveillance activity has dramatically increased over the past few years. Earlier work dealt mostly with single stationary cameras, but the recent trend is toward active multicamera systems. Such systems offer several advantages over single camera systems - multiple overlapping views for obtaining 3D information and handling occlusions, multiple nonoverlapping cameras for covering wide areas, and active pan-tilt-zoom (PTZ) cameras for observing object details. To address these issues, we have developed a multicamera video surveillance approach, called distributed interactive video array. The DIVA framework provides multiple levels of semantically meaningful information ("situational" awareness) to match the needs of multiple remote observers. We have designed DIVA-based systems that can track and identify vehicles and people, monitor perimeters and bridges, and analyze activities. A new video surveillance approach employing a large-scale cluster of video sensors demonstrates the promise of multicamera arrays for homeland security.

Anatomy of a multicamera video surveillance system

We present a framework for multicamera video surveillance. The framework consists of three phases: detection, representation, and recognition. The detection phase handles multisource spatiotemporal data fusion for efficiently and reliably extracting motion trajectories from video. The representation phase summarizes raw trajectory data to construct hierarchical, invariant, and content-rich descriptions of the motion events. Finally, the recognition phase deals with event classification and identification on the data descriptors. Through empirical study in a parking-lot-surveillance setting, we show that our spatiotemporal fusion scheme and biased sequence-data learning method are highly effective in identifying suspicious events.