Multi-camera Network Research Articles

Validation of Camera Networks Used for the Assessment of Speech Movements

Abstract. The term speech sound disorder describes a range of speech difficulties in children that affect speech intelligibility. Differential diagnosis is difficult and reliant on access to validated and reliable measures. Technological advances aim to provide clinical access to measurements that have been identified as beneficial in diagnosing speech disorders. To generate objective measurements and, consequently, automatic scores, the output from multi-camera networks is required to produce quality results. The quality of photogrammetric results is usually expressed in terms of the precision and reliability of the network. Precision is determined at the design stage as a function of the geometry of the network. In this manuscript, we focus on the design of a photogrammetric camera network using three cameras. We adopted a similar workflow as Alsadika et al. (2012) and tested serval network configurations. As the distances from the camera stations to object points were fixed to 3500mm, only the horizontal and vertical placements of the cameras were varied. Horizontal angles were changed within an increment of 10º, and vertical angles were changed within an increment of 5º. The object space coordinates of GCPs for each camera configuration were assessed in terms of horizontal error ellipses and vertical precision. The best design was the maximum horizontal and vertical convergence angles of 90° and 30°. The existing camera network used to capture videos for speech assessment was approximately as good as the top third of tested designs. However, from a validation perspective, it can be concluded that the design is viable for continued use.

A comprehensive survey on machine learning techniques to mobilize multi-camera network for smart surveillance

A comprehensive survey on machine learning techniques to mobilize multi-camera network for smart surveillance

Person re-identification: A retrospective on domain specific open challenges and future trends

Person Re-Identification (Re-ID) is a critical aspect of visual surveillance systems, which aims to automatically recognize and locate individuals across a multi-camera network with non-overlapping fields-of-view. Despite significant progress in recent years through the use of deep learning-based approaches, there remain many vision-related challenges, such as occlusion, pose, background clutter, misalignment, scale, viewpoint, low resolution & illumination, and cross-domain generalization across camera modalities, that hinder the accurate identification of individuals. The majority of the proposed approaches directly or indirectly aim to solve one or multiple of these existing challenges. To further advance the development of Re-ID solutions, a comprehensive review of the current approaches is necessary. However, no focused review currently exists that analyses and highlights specific aspects for further development. To fill this gap, we present a systematic challenge-specific literature survey of about 300 papers published between 2015 and 2022, which reviews Re-ID approaches from a solution-oriented perspective. This survey is the first of its kind to provide an in-depth analysis of the different approaches used to address the various challenges in Re-ID. Furthermore, our review highlights several prominent and diverse research trends in the Re-ID domain. These trends offer a visionary perspective regarding ongoing person Re-ID research, and they may eventually lead to the development of practical real-world solutions. We highlighted the AI ethics that must be followed while developing a Re-ID solution, and recently being practiced as well. Another exciting future dimension of person Re-ID research is the long-term Re-ID, which is still under evolution. Overall, our survey aims to serve as a valuable resource for researchers and practitioners working in the field of Re-ID and to inspire the development of innovative and effective Re-ID solutions.

Language and vision based person re-identification for surveillance systems using deep learning with LIP layers

Real-time surveillance systems have become a necessity of today's life owing to their relevance in the contemporary era for security reasons to ensure a secure and safe environment. Presently, Person re-identification (Re-ID)-based surveillance systems are becoming increasingly more prevalent and sophisticated since they do not require human intervention and are more reliable to deploy in public spaces leveraging multi-camera networks. However, one of the major problems in Person ReID is the visual appearance i-e the appearance of a person in an image is greatly affected by different camera views. As a result, the discriminative set of features must be learned in a deep learning model in order to re-identify persons from opposing camera viewpoints. To address this challenge, we propose an image/text-retrieval-based Person ReId method in which both visual and text-based features are exploited to carry out person re-identification. More precisely, the textual descriptions of the images are taken into account as text features with Glove Word Embedding followed by 1D-MAPCNN and fused with image-level features extracted using the GoogLeNet model. In addition, the feature discriminability is enhanced using local importance-based pooling (LIP) layers in which adaptive significance weights are learned during downsampling. Moreover, from two different modalities, feature refinement is done during training with the help of attention mechanisms using the Convolutional Block Attention module (CBAM) and the proposed shared attention neural network. It is observed that LIP layers along with both vision and textual features are playing a key role in acquiring discriminative features even if the visual appearance of the same person is greatly affected due to camera pose conditions. The proposed method is validated on the CUHK-PADES dataset and has 15.34% and 24.39% rank-1 improvement in text and image-based retrievals.

A Nonoverlapping Vision Field Multi-Camera Network for Tracking Human Build Targets

This research presents a procedure for tracking human build targets in a multi-camera network with nonoverlapping vision fields. The proposed approach consists of three main steps: single-camera target detection, single-camera target tracking, and multi-camera target association and continuous tracking. The multi-camera target association includes target characteristic extraction and the establishment of topological relations. Target characteristics are extracted based on the HSV (Hue, Saturation, and Value) values of each human build movement target, and the space-time topological relations of the multi-camera network are established using the obtained target associations. This procedure enables the continuous tracking of human build movement targets in large scenes, overcoming the limitations of monitoring within the narrow field of view of a single camera.

Evaluation of Camera Pose Estimation Using Human Head Pose Estimation

We introduce and evaluate a novel camera pose estimation framework that uses the human head as a calibration object. The proposed method facilitates extrinsic calibration from 2D input images (NIR and/or RGB), while merely relying on the detected human head, without the need for depth information. The approach is applicable to single cameras or multi-camera networks. Our implementation uses a fine-tuned deep learning-based 2D human facial landmark detector to estimate the 3D human head pose by fitting a 3D head model to the detected 2D facial landmarks. Our work focuses on an evaluation of the proposed approach on real multi-camera recordings and synthetic renderings to determine the accuracy of the pose estimation results and their applicability. We assess the robustness of our method against different input parameters, such as varying relative camera positions, variations of head models, face occlusions (by masks, sun glasses, etc.), potential biases and variance among humans. Based on the experimental results, we expect our approach to be effective for numerous use cases including automotive attention monitoring, robotics, VR/AR and other scenarios where ease of handling outweighs accuracy.

Parallel optimization of tridimensional deformation measurement based on correlation function constraints of a multi-camera network.

This study primarily investigates the low accuracy and redundant time-consuming problem of speckle registration in the full-field deformation measurement of slender and biggish specimens. To solve these problems, a parallel optimization of the tridimensional deformation measurement method is proposed based on what we believe is a novel correlation function constraints of a multi-camera network. First, a neotype correlation function is built based on the joint constraint relationship among the multiple cameras, which is capable of accurately restricting the search for homologous points in image pairs to the epipolar line, instead of the entire image, while significantly narrowing the search space and accelerating the search. The multiple cameras are bundled as a whole, thus reducing the dimension of the Jacobian matrix and the normalized matrix to a certain extent. Subsequently, more speckle images can be calculated in one iteration. Furthermore, the decomposition of the derived correlation function and the scheme of the parallel algorithm are decomposed via the kernel function based on the GPU parallel mechanism of the compute unified device architecture source program, thus increasing the subpixel search speed of speckle matching and ensuring the calculation performance of the stereo deformation measurement method to reach a higher level. Lastly, the experimental results revealed that the proposed strategy could allow the calculation speed-up ratio of speckle sequence and stereo registration to reach 20.390 times and 17.873 times, respectively, while ensuring the out-of-plane displacement average measuring accuracy to be higher than 0.179mm within the spatial range of [2m, 2m, 3m]. As a result, the proposed approach has crucial applications in rapid and stable tridimensional deformation measurement.

Lightweight Indoor Multi-Object Tracking in Overlapping FOV Multi-Camera Environments.

Multi-Target Multi-Camera Tracking (MTMCT), which aims to track multiple targets within a multi-camera network, has recently attracted considerable attention due to its wide range of applications. The main challenge of MTMCT is to match local tracklets (i.e., sub-trajectories) obtained by different cameras and to combine them into global trajectories across the multi-camera network. This paper addresses the cross-camera tracklet matching problem in scenarios with partially overlapping fields of view (FOVs), such as indoor multi-camera environments. We present a new lightweight matching method for the MTMC task that employs similarity analysis for location features. The proposed approach comprises two steps: (i) extracting the motion information of targets based on a ground projection method and (ii) matching the tracklets using similarity analysis based on the Dynamic Time Warping (DTW) algorithm. We use a Kanade–Lucas–Tomasi (KLT) algorithm-based frame-skipping method to reduce the computational overhead in object detection and to produce a smooth estimate of the target’s local tracklets. To improve matching accuracy, we also investigate three different location features to determine the most appropriate feature for similarity analysis. The effectiveness of the proposed method has been evaluated through real experiments, demonstrating its ability to accurately match local tracklets.

Shareability-Exclusivity Representation on Product Grassmann Manifolds for Multi-camera video clustering

With the rapid popularity of multi-camera networks, one human action is usually captured by multiple cameras located at different angles simultaneously. Multi-camera videos contain the distinct perspectives of one action, therefore multiple views can overcome the impacts of illumination and occlusion. In this paper, we propose a novel multi-camera video clustering model, named Shareability-Exclusivity Representation on Product Grassmann Manifolds (PGM-SER), to address two key issues in traditional multi-view clustering methods (MVC): (1) Most MVC methods directly construct a shared similarity matrix by fusing multi-view data or their corresponding similarity matrices, which ignores the exclusive information in each view; (2) Most MVC methods are designed for multi-view vectorial data, which cannot handle the nonlinear manifold structure hidden in multi-camera videos. The proposed PGM-SER firstly adopts Product Grassmann Manifolds to represent multi-camera videos, then simultaneously learn their shared and exclusive information in global structures to achieve multi-camera video clustering. We provide an effective optimization algorithm to solve PGM-SER and present the corresponding convergence analysis. Finally, PGM-SER is tested on three multi-camera human action video datasets and obtain satisfied experimental results.

Efficient and Consumer-Centered Item Detection and Classification with a Multicamera Network at High Ranges.

In the EU project SHAREWORK, methods are developed that allow humans and robots to collaborate in an industrial environment. One of the major contributions is a framework for task planning coupled with automated item detection and localization. In this work, we present the methods used for detecting and classifying items on the shop floor. Important in the context of SHAREWORK is the user-friendliness of the methodology. Thus, we renounce heavy-learning-based methods in favor of unsupervised segmentation coupled with lenient machine learning methods for classification. Our algorithm is a combination of established methods adjusted for fast and reliable item detection at high ranges of up to eight meters. In this work, we present the full pipeline from calibration, over segmentation to item classification in the industrial context. The pipeline is validated on a shop floor of 40 sqm and with up to nine different items and assemblies, reaching a mean accuracy of 84% at Hz.

Generalized multiple sparse information fusion for vehicle re-identification

Vehicle re-identification (reID) aims to search the target vehicle in a non-overlapping multi-camera network, which is important for the intelligent analysis in large scale of surveillance videos. Many existing methods have employed various techniques to achieve discriminative information. However, those methods always focus on the description of one view for the same vehicle images. Hence, a generated multiple sparse information fusion method is proposed for exploiting latent features from multi-views, which employs three different deep networks to extract multiple features from coarse to fine. And these features are regarded as multi-view features. Besides, to fuse these features reasonably, the paper transfers various features into a common space for better seeking distinctive features. Especially, besides ResNet, two feature learning networks are proposed to learn different features, respectively. One is designed to learn robust feature by dropping some features randomly when training the reID model. Another is to combine various salient features from different layers, which forms strong features for the reID task. Moreover, comprehensive experimental results have demonstrated that our proposed method can achieve competitive performances on benchmark datasets VehicleID and VeRi-776.

Real-Time In-Network Image Compression via Distributed Dictionary Learning

Multi-camera networks are increasingly becoming pervasive in many monitoring and surveillance applications, and have attracted much attention in distributed systems with collaborative, real-time decision-making capabilities. While in-network data compression brings significant energy savings in camera nodes, signal representation using sparse approximations and overcomplete dictionaries have been shown to outperform traditional compression methods. In this work, an end-to-end and real-time solution is designed and implemented to enable energy-efficient and robust dictionary learning in distributed camera networks by leveraging the spatial correlation of the collected multimedia data. Traditional distributed dictionary learning relies on consensus-building algorithms, which involve communicating with neighboring nodes until convergence is achieved. Existing methods, however, do not exploit spatial correlations in camera networks for improved energy efficiency. In contrast, low-computational-complexity metrics are employed in this work to quantify and exploit the spatial correlation across camera nodes in a wireless network for efficient distributed dictionary learning and in-network image compression. The performance of the proposed approach is validated through extensive simulations on public datasets as well as via real-world experiments on a testbed composed of Raspberry Pi nodes.

Radial Coverage Strength for Optimization of Monocular Multicamera Deployment

In this article, a new concept, radial coverage strength, is first proposed to characterize the visual sensing performance when the orientation of the target pose is considered. In particular, the elevation angle of the optical pose of the visual sensor is taken to decompose the visual coverage strength into effective and ineffective components, motivated by the imaging intuition. An optimization problem is then formulated for a monocular multicamera network to maximize the coverage of the object area based on the strength information fusion along the effective coverage strength direction through the deployment of the angle between radial coverage vector of the camera optical pose. Both simulation and experiments are conducted to validate the proposed approach and comparison with existing methods is also provided.

High-precision and full-field measurement of 3D deformation based on multi-camera network joint constraint optimization

High-precision and full-field measurement of 3D deformation based on multi-camera network joint constraint optimization

Maximizing Camera Coverage in Multicamera Surveillance Networks

Efficient placement of cameras to perform surveillance tasks has a significant impact on the overall performance as well as cost of video surveillance systems. Identifying an optimal configuration for cameras in a surveillance system to maximize coverage is a combinatorial optimization problem. This paper proposes two algorithms for identifying optimal camera configuration for a multi-camera network with predefined camera locations. We propose an Alternate Global Greedy (AGG) algorithm, a novel variant of the Global Greedy algorithm by making significant modifications to the traditional execution strategy to produce better coverage results. Additionally, we propose an innovative Greedy Grid Voting (GGV) algorithm, which gives preference to cover unique as well as critical regions and effectively addresses a broad range of coverage scenarios mandated by the application. The proposed algorithms are validated on map images across diverse scenarios with predefined camera locations. Experimental results show that our algorithms maximize the coverage results with minimal overlap, compared to the existing greedy techniques.

Introducing Scene Understanding to Person Re-Identification using a Spatio-Temporal Multi-Camera Model

In this paper, we investigate person re-identification (re-ID) in a multi-camera network for surveillance applications. To this end, we create a Spatio-Temporal Multi-Camera model (ST-MC model), which exploits statistical data on a person’s entry/exit points in the multi-camera network, to predict in which camera view a person will re-appear. The created ST-MC model is used as a novel extension to the Multiple Granularity Network (MGN) [1], which is the current state of the art in person re-ID. Compared to existing approaches that are solely based on Convolutional Neural Networks (CNNs), our approach helps to improve the re-ID performance by considering not only appearance-based features of a person from a CNN, but also contextual information. The latter serves as scene understanding information complimentary to person re-ID. Experimental results show that for the DukeMTMC-reID dataset [2][3], introduction of our ST-MC model substantially increases the mean Average Precision (mAP) and Rank-1 score from 77.2% to 84.1%, and from 88.6% to 96.2%, respectively.

Multi-Target Multi-Camera Tracking by Tracklet-to-Target Assignment

This paper focuses on the Multi-Target Multi-Camera Tracking task (MTMCT), which aims at tracking multiple targets within a multi-camera network. As the trajectory of each target is inherently split into multiple sub-trajectories (namely local tracklets) in a multi-camera network, a major challenge of MTMCT is how to accurately match the local tracklets generated within each camera across different cameras and generate a complete global trajectory for each target, i.e. , the cross-camera tracklet matching problem. We solve the cross-camera tracklet matching problem by TRACklet-to-Target Assignment (TRACTA), and propose the Restricted Non-negative Matrix Factorization (RNMF) algorithm to compute the optimal assignment solution that meets a set of constraints, which should be in force in practice. TRACTA can correct the tracking errors caused by occlusions and missed detections in local tracklets, and produce a complete global trajectory for each target across all the cameras. Moreover, we also develop an analytical way of estimating the total number of targets in the camera network, which plays an important role to compute the tracklet-to-target assignment. Experimental evaluations and ablation studies on four MTMCT benchmark datasets show the superiority of the proposed TRACTA method.

Multi-view Person Re-identification in a Fisheye Camera Network with Different Viewing Directions

Person re-identification is a crucial component for multi-camera networks in different real-world applications such as surveillance, automation, and business analytics. Despite considerable recent progress, the performance in practice is still not satisfactory due to the high intra-person variation and significant complexity of the task, including differences in scale, viewing direction, and illumination. We propose a novel approach for person re-identification, which exploits multi-view information of a fisheye camera looking downwards from the ceiling. To handle this highly variable multi-view information, we build a generic pipeline for processing fisheye camera imagery based on geometric sensor modelling and deep learning. The proposed approach is evaluated on a re-mapped version of the publicly available Market-1501 dataset, and, in addition, on a new fisheye dataset. Significant improvements are shown in our experiments: our approach achieves more than 97% rank#1 recognition rate if applied on the re-mapped Market-1501 dataset; on the new fisheye dataset we find an improvement of about 12% compared to random-view fusion.

View's dependency and low‐rank background‐guided compressed sensing for multi‐view image joint reconstruction

Compressed sensing (CS) multi-camera network reconstruction has attracted much attention in the field of distributed CS networks. However, many multi-camera network reconstructions based on CS usually recover every image separately; the view's dependency and geometrical structure among these multi-view images could be rarely considered in this way, which will result in some unsatisfied joint reconstruction results. Here, the authors introduce to extract the multiple view geometry from multi-view images to construct the view's dependency observation model. Based on the proposed parametric transformation observation model, they propose a novel CS joint reconstruction method of multi-view image that guided by the spatial correlation and low-rank background constraints. The eventual optimisation model could be relaxed to a series of convex optimisation problems, which could be efficiently solved by combining the variable splitting and alternate iteration technique. The extended experimental results indicate that they proposed method has achieved a remarkable improvement in both objective criterion and visual fidelity compared with other competitive reconstruction methods.

Dense 3D-Convolutional Neural Network for Person Re-Identification in Videos

Person re-identification aims at identifying a certain pedestrian across non-overlapping multi-camera networks in different time and places. Existing person re-identification approaches mainly focus on matching pedestrians on images; however, little attention has been paid to re-identify pedestrians in videos. Compared to images, video clips contain motion patterns of pedestrians, which is crucial to person re-identification. Moreover, consecutive video frames present pedestrian appearance with different body poses and from different viewpoints, providing valuable information toward addressing the challenge of pose variation, occlusion, and viewpoint change, and so on. In this article, we propose a Dense 3D-Convolutional Network (D3DNet) to jointly learn spatio-temporal and appearance representation for person re-identification in videos. The D3DNet consists of multiple three-dimensional (3D) dense blocks and transition layers. The 3D dense blocks enlarge the receptive fields of visual neurons in both spatial and temporal dimensions, leading to discriminative appearance representation as well as short-term and long-term motion patterns of pedestrians without the requirement of an additional motion estimation module. Moreover, we formulate a loss function consisting of an identification loss and a center loss to minimize intra-class variance and maximize inter-class variance simultaneously, toward addressing the challenge of large intra-class variance and small inter-class variance. Extensive experiments on two real-world video datasets of person identification, i.e., MARS and iLIDS-VID, have shown the effectiveness of the proposed approach.