Multi-camera Setting Research Articles

Multi-camera multi-object tracking on the move via single-stage global association approach

The development of autonomous vehicles generates a tremendous demand for a low-cost solution based on a complete set of camera sensors to perceive the environment around the car. Towards this solution, it is essential for object detection and tracking to address new challenges in multi-camera settings. To address these challenges, this work introduces novel Single-Stage Global Association Tracking approaches to associate one or more detections from multi-cameras with tracked objects. These approaches aim to solve fragment-tracking issues caused by inconsistent 3D object detection. Moreover, our models also improve the detection accuracy of the standard vision-based 3D object detectors in the nuScenes detection challenge. The extensive experimental results on the nuScenes dataset demonstrate the benefits of the proposed method, which outperforms prior vision-based tracking methods in multi-camera settings.

Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving

Multi-camera perception tasks have gained significant attention in the field of autonomous driving. However, existing frameworks based on Lift-Splat-Shoot (LSS) in the multi-camera setting cannot produce suitable dense 3D features due to the projection nature and uncontrollable densification process. To resolve this problem, we propose to regulate intermediate dense 3D features with the help of volume rendering. Specifically, we employ volume rendering to process the dense 3D features to obtain corresponding 2D features (e.g., depth maps, semantic maps), which are supervised by associated labels in the training. This manner regulates the generation of dense 3D features on the feature level, providing appropriate dense and unified features for multiple perception tasks. Therefore, our approach is termed Vampire, stands for ``Volume rendering As Multi-camera Perception Intermediate feature REgulator''. Experimental results on the Occ3D and nuScenes datasets demonstrate that Vampire facilitates fine-grained and appropriate extraction of dense 3D features, and is competitive with existing SOTA methods across diverse downstream perception tasks like 3D occupancy prediction, LiDAR segmentation and 3D objection detection, while utilizing moderate GPU resources. We provide a video demonstration in the supplementary materials and Codes are available at github.com/cskkxjk/Vampire.

Multi-camera multi-object tracking: A review of current trends and future advances

The nascent applicability of multi-camera tracking (MCT) in numerous real-world applications makes it a significant computer vision problem. While visual tracking of objects, especially in video obtained from single camera setup, has drawn huge research attention, the constant identification and tracking of targets as they transit across multiple cameras remains an open research problem. In addition to the linking of target appearance and trajectory information across frames, effective association of such data across multiple cameras is also very critical in MCT. Occlusion, appearance variability, camera motion, as well as nonrigid object structure and motion are widely recognized constraints and major sources of concerns in MCT. In recent years, several literatures have been contributed suggesting a variety of approaches to addressing various problems in MCT. However, studies that critically review and report the advances and trends of research in MCT are still limited. This current study presents a comprehensive and up-to-date review of visual object tracking in multi-camera settings. In this paper, we analyze and categorize existing works based on six crucial facets: problem formulation, adopted problem solving approach, data association requirements, mutual exclusion constraints, benchmark datasets, and performance metrics. Furthermore, the study summarizes the outcomes of 30 state-of-the-art MCT algorithms on common datasets to allow quantitative comparison and analysis of their experimental results. Finally, we examine recent advances in MCT and suggest some promising future research directions.

Transfer Learning for Color Constancy via Statistic Perspective

Color Constancy aims to correct image color casts caused by scene illumination. Recently, although the deep learning approaches have remarkably improved on single-camera data, these models still suffer from the seriously insufficient data problem, resulting in shallow model capacity and degradation in multi-camera settings. In this paper, to alleviate this problem, we present a Transfer Learning Color Constancy (TLCC) method that leverages cross-camera RAW data and massive unlabeled sRGB data to support training. Specifically, TLCC consists of the Statistic Estimation Scheme (SE-Scheme) and Color-Guided Adaption Branch (CGA-Branch). SE-Scheme builds a statistic perspective to map the camera-related illumination labels into camera-agnostic form and produce pseudo labels for sRGB data, which greatly expands data for joint training. Then, CGA-Branch further promotes efficient transfer learning from sRGB to RAW data by extracting color information to regularize the backbone's features adaptively. Experimental results show the TLCC has overcome the data limitation and model degradation, outperforming the state-of-the-art performance on popular benchmarks. Moreover, the experiments also prove the TLCC is capable of learning new scenes information from sRGB data to improve accuracy on the RAW images with similar scenes.

Full Surround Monodepth From Multiple Cameras

Self-supervised monocular depth and ego-motion estimation is a promising approach to replace or supplement expensive depth sensors such as LiDAR for robotics applications like autonomous driving. However, most research in this area focuses on a single monocular camera or stereo pairs that cover only a fraction of the scene around the vehicle. In this work, we extend monocular self-supervised depth and ego-motion estimation to large-baseline multi-camera rigs. Using generalized spatio-temporal contexts, pose consistency constraints, and carefully designed photometric loss masking, we learn a single network generating dense, consistent, and scale-aware point clouds that cover the same full surround <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$360^{\circ }$</tex-math></inline-formula> field of view as a typical LiDAR scanner. We also propose a new scale-consistent evaluation metric more suitable to multi-camera settings. Experiments on two challenging benchmarks illustrate the benefits of our approach over strong baselines.

Instance Segmentation with Mask R-CNN Applied to Loose-Housed Dairy Cows in a Multi-Camera Setting.

Simple SummaryAs sociability in cattle is a matter of good animal husbandry, this study provides technical groundwork for a camera based system to automatically analyse dairy cattle herd activity. Eight surveillance cameras were recording a group of thirty-six lactating Holstein Friesian dairy cows. A Mask R-CNN model was trained to determine pixel level segmentation masks for the cows in the video material. The animals were successfully segmented reaching high ‘averaged precision scores’ for bounding boxes (0.91) and segmentation masks (0.85) for a given IOU threshold of 0.5. As providing training data for deep learning models is time consuming and tedious, this article also deals with the question “How many images do I have to annotate?” and analyses the performance of the model depending on the size of the used training data set.With increasing herd sizes came an enhanced requirement for automated systems to support the farmers in the monitoring of the health and welfare status of their livestock. Cattle are a highly sociable species, and the herd structure has important impact on the animal welfare. As the behaviour of the animals and their social interactions can be influenced by the presence of a human observer, a camera based system that automatically detects the animals would be beneficial to analyse dairy cattle herd activity. In the present study, eight surveillance cameras were mounted above the barn area of a group of thirty-six lactating Holstein Friesian dairy cows at the Chamber of Agriculture in Futterkamp in Northern Germany. With Mask R-CNN, a state-of-the-art model of convolutional neural networks was trained to determine pixel level segmentation masks for the cows in the video material. The model was pre-trained on the Microsoft common objects in the context data set, and transfer learning was carried out on annotated image material from the recordings as training data set. In addition, the relationship between the size of the used training data set and the performance on the model after transfer learning was analysed. The trained model achieved averaged precision (Intersection over union, IOU = 0.5) 91% and 85% for the detection of bounding boxes and segmentation masks of the cows, respectively, thereby laying a solid technical basis for an automated analysis of herd activity and the use of resources in loose-housing.

Pose-Assisted Multi-Camera Collaboration for Active Object Tracking

Active Object Tracking (AOT) is crucial to many vision-based applications, e.g., mobile robot, intelligent surveillance. However, there are a number of challenges when deploying active tracking in complex scenarios, e.g., target is frequently occluded by obstacles. In this paper, we extend the single-camera AOT to a multi-camera setting, where cameras tracking a target in a collaborative fashion. To achieve effective collaboration among cameras, we propose a novel Pose-Assisted Multi-Camera Collaboration System, which enables a camera to cooperate with the others by sharing camera poses for active object tracking. In the system, each camera is equipped with two controllers and a switcher: The vision-based controller tracks targets based on observed images. The pose-based controller moves the camera in accordance to the poses of the other cameras. At each step, the switcher decides which action to take from the two controllers according to the visibility of the target. The experimental results demonstrate that our system outperforms all the baselines and is capable of generalizing to unseen environments. The code and demo videos are available on our website https://sites.google.com/view/pose-assisted-collaboration.

Automatic classification of dopamine transporter SPECT: deep convolutional neural networks can be trained to be robust with respect to variable image characteristics.

This study investigated the potential of deep convolutional neural networks (CNN) for automatic classification of FP-CIT SPECT in multi-site or multi-camera settings with variable image characteristics. The study included FP-CIT SPECT of 645 subjects from the Parkinson's Progression Marker Initiative (PPMI), 207 healthy controls, and 438 Parkinson's disease patients. SPECT images were smoothed with an isotropic 18-mm Gaussian kernel resulting in 3 different PPMI settings: (i) original (unsmoothed), (ii) smoothed, and (iii) mixed setting comprising all original and all smoothed images. A deep CNN with 2,872,642 parameters was trained, validated, and tested separately for each setting using 10 random splits with 60/20/20% allocation to training/validation/test sample. The putaminal specific binding ratio (SBR) was computed using a standard anatomical ROI predefined in MNI space (AAL atlas) or using the hottest voxels (HV) analysis. Both SBR measures were trained (ROC analysis, Youden criterion) using the same random splits as for the CNN. CNN and SBR trained in the mixed PPMI setting were also tested in an independent sample from clinical routine patient care (149 with non-neurodegenerative and 149 with neurodegenerative parkinsonian syndrome). Both SBR measures performed worse in the mixed PPMI setting compared to the pure PPMI settings (e.g., AAL-SBR accuracy = 0.900 ± 0.029 in the mixed setting versus 0.957 ± 0.017 and 0.952 ± 0.015 in original and smoothed setting, both p < 0.01). In contrast, the CNN showed similar accuracy in all PPMI settings (0.967 ± 0.018, 0.972 ± 0.014, and 0.955 ± 0.009 in mixed, original, and smoothed setting). Similar results were obtained in the clinical sample. After training in the mixed PPMI setting, only the CNN provided acceptable performance in the clinical sample. These findings provide proof of concept that a deep CNN can be trained to be robust with respect to variable site-, camera-, or scan-specific image characteristics without a large loss of diagnostic accuracy compared with mono-site/mono-camera settings. We hypothesize that a single CNN can be used to support the interpretation of FP-CIT SPECT at many different sites using different acquisition hardware and/or reconstruction software with only minor harmonization of acquisition and reconstruction protocols.

The Added Value of Studying Embodied Responses in Couple Therapy Research: A Case Study.

This article reports on the added value of embodied responses identified through sympathetic nervous system (SNS) activity in couple therapy research. It focuses on moments of change and the timing of therapeutic interventions or therapeutic moves in a couple therapy session. The data for this single-case study comprise couple therapy process videotapes recorded in a multi-camera setting, and measurements of participants' SNS activity. The voluntary participants were a marital couple in their late thirties and two middle-aged male psychotherapists. The division into topic segments showed how the key issue of seeking help, which was found to comprise three separate components, was repeatedly dealt with in the session. SNS activity showed different degrees of synchronization between the couple, between the therapists, and between the couple and therapists during the dialogue pertaining to these three components. The issue of timing emerged as a complex, even ambivalent, phenomenon. Arousal in the therapists was in line with their therapeutic activity, whereas in the clients it was more anticipatory. The approach used here rendered visible some of the intensity that therapeutic dialogue can generate when dealing with issues of relationship change in the couple context and showed how this intensity can be dialogically regulated in the therapeutic system.

Do nursery habitats provide shelter from flow for juvenile fish?

Juvenile fish nurseries are an essential life stage requirement for the maintenance of many fish populations. With many inshore habitats globally in decline, optimising habitat management by increasing our understanding of the relationship between juvenile fish and nursery habitats may be a prudent approach. Previous research on post–settlement snapper (Chrysophrys auratus) has suggested that structure may provide a water flow refuge, allowing snapper to access high water flow sites that will also have a high flux of their pelagic prey. We investigated this hypothesis by describing how Artificial Seagrass Units (ASUs) modified water flow while also using a multi–camera set up to quantify snapper position in relation to this water flow environment. Horizontal water flow was reduced on the down–current side of ASUs, but only at the height of the seagrass canopy. While the highest abundance of snapper did occur down–current of the ASUs, many snapper also occupied other locations or were too high in the water column to receive any refuge from water flow. The proportion of snapper within the water column was potentially driven by strategy to access zooplankton prey, being higher on the up–current side of ASUs and on flood tides. It is possible that post–settlement snapper alternate position to provide opportunities for both feeding and flow refuging. An alternative explanation relates to an observed interaction between post–settlement snapper and a predator, which demonstrated that snapper can utilise habitat structure when threatened. The nature of this relationship, and its overall importance in determining the value of nursery habitats to post–settlement snapper remains an elusive next step.

Attributes driven tracklet-to-tracklet person re-identification using latent prototypes space mapping

Most of current person re-identification works identify a person by matching his/her probe image against a galley images set. One feasible way to improve the identification accuracy is the multi-shot re-identification, where the probe includes a small set of images rather than a single image. In this paper, we study the tracklet-to-tracklet identification, where both the probe and the target dataset are composed of small sets of sequential images, i.e., tracklets. To solve this problem and make our algorithm robust under multi-camera setting, we take full advantage of low-level features, attributes and inter-attribute correlations at the same time. Attributes are expected to offer semantic information complementary to low-level features. In order to discover the correlations among attributes, a novel discriminative model is proposed to exploit low-level features and map attributes to a discriminative latent prototypes space. An alternating optimization procedure is designed to perform the learning process. We also devise a number of voting schemes to total up matching scores from images to tracklets. Experiments on four public datasets show that our approach achieves a consistently better performance than existing person re-identification methods.

Reduction in camera-specific variability in [(123)I]FP-CIT SPECT outcome measures by image reconstruction optimized for multisite settings: impact on age-dependence of the specific binding ratio in the ENC-DAT database of healthy controls.

Quantitative estimates of dopamine transporter availability, determined with [(123)I]FP-CIT SPECT, depend on the SPECT equipment, including both hardware and (reconstruction) software, which limits their use in multicentre research and clinical routine. This study tested a dedicated reconstruction algorithm for its ability to reduce camera-specific intersubject variability in [(123)I]FP-CIT SPECT. The secondary aim was to evaluate binding in whole brain (excluding striatum) as a reference for quantitative analysis. Of 73 healthy subjects from the European Normal Control Database of [(123)I]FP-CIT recruited at six centres, 70 aged between 20 and 82years were included. SPECT images were reconstructed using the QSPECT software package which provides fully automated detection of the outer contour of the head, camera-specific correction for scatter and septal penetration by transmission-dependent convolution subtraction, iterative OSEM reconstruction including attenuation correction, and camera-specific "to kBq/ml" calibration. LINK and HERMES reconstruction were used for head-to-head comparison. The specific striatal [(123)I]FP-CIT binding ratio (SBR) was computed using the Southampton method with binding in the whole brain, occipital cortex or cerebellum as the reference. The correlation between SBR and age was used as the primary quality measure. The fraction of SBR variability explained by age was highest (1) with QSPECT, independently of the reference region, and (2) with whole brain as the reference, independently of the reconstruction algorithm. QSPECT reconstruction appears to be useful for reduction of camera-specific intersubject variability of [(123)I]FP-CIT SPECT in multisite and single-site multicamera settings. Whole brain excluding striatal binding as the reference provides more stable quantitative estimates than occipital or cerebellar binding.

MoVieUp: Automatic Mobile Video Mashup

With the proliferation of mobile devices, people are taking videos of the same events anytime and anywhere. Even though these crowdsourced videos are uploaded to the cloud and shared, the viewing experience is very limited due to monotonous viewing, visual redundancy, and bad audio-video quality. In this paper, we present a fully automatic mobile video mashup system that works in the cloud to combine recordings captured by multiple devices from different view angles and at different time slots into a single yet enriched and professional looking video-audio stream. We summarize a set of computational filming principles for multicamera settings from a formal focus study. Based on these principles, given a set of recordings of the same event, our system is able to synchronize these recordings with audio fingerprints, assess audio and video quality, detect video cut points, and generate video and audio mashups. The audio mashup is the maximization of audio quality under the less switching principle, while the video mashup is formalized as maximizing video quality and content diversity, constrained by the summarized filming principles. Our system is different from any existing work in this field in three ways: 1) our system is fully automatic; 2) the system incorporates a set of computational domain-specific filming principles summarized from a formal focus study; and 3) in addition to video, we also consider audio mashup that is a key factor of user experience (UX) yet often overlooked in existing research. Evaluations show that our system achieves performance results that are superior to state-of-the-art video mashup techniques, thus providing a better UX.

People and Mobile Robot Classification Through Spatio-Temporal Analysis of Optical Flow

The goal of this work is to distinguish between humans and robots in a mixed human-robot environment. We analyze the spatio-temporal patterns of optical flow-based features along several frames. We consider the Histogram of Optical Flow (HOF) and the Motion Boundary Histogram (MBH) features, which have shown good results on people detection. The spatio-temporal patterns are composed of groups of feature components that have similar values on previous frames. The groups of features are fed into the FuzzyBoost algorithm, which at each round selects the spatio-temporal pattern (i.e. feature set) having the lowest classification error. The search for patterns is guided by grouping feature dimensions, considering three algorithms: (a) similarity of weights from dimensionality reduction matrices, (b) Boost Feature Subset Selection (BFSS) and (c) Sequential Floating Feature Selection (SFSS), which avoid the brute force approach. The similarity weights are computed by the Multiple Metric Learning for large Margin Nearest Neighbor (MMLMNN), a linear dimensionality algorithm that provides a type of Mahalanobis metric Weinberger and Saul, J. MaCh. Learn. Res.10 (2009) 207–244. The experiments show that FuzzyBoost brings good generalization properties, better than the GentleBoost, the Support Vector Machines (SVM) with linear kernels and SVM with Radial Basis Function (RBF) kernels. The classifier was implemented and tested in a real-time, multi-camera dynamic setting.

Single and Multi Camera Simultaneous Localization and Mapping Using the Extended Kalman Filter

Simultaneous Localization and Mapping (SLAM) has received quite a lot of attention in the last decades because of its relevance for many applications centered on a mobile observer, such as service robotics and intelligent transportation systems. This paper focuses on the use of recursive Bayesian filtering, as implemented by the Extendend Kalman Filter (EKF), to face the Visual SLAM problem, i.e., when using data from visual sources. In Monocular SLAM, which uses a single camera as unique source of information, it is not possible to directly estimate the depth of a feature from a single image. To handle the severely non-normal distribution representing such uncertainty, inverse parametrizations were developed, capable to deal with such uncertainty and still relying on Gaussian variables. In the paper, after an introduction to EKF-SLAM, we provide a review of different inverse parametrizations, and we introduce a novel proposal, the Framed Inverse Depth (FID) parametrization, which, in terms of consistency, performs similarly to state of the art Monocular SLAM parametrizations, but at a reduced computational cost. All these parametrizations can be used in a stereo and multi camera setting too. An extensive analysis is presented for both Monocular and stereo SLAM, for a simulated environment widely used in the literature as well as on a widely used real dataset.

Region-based pose tracking with occlusions using 3D models

Despite great progress achieved in 3-D pose tracking during the past years, occlusions and self-occlusions are still an open issue. This is particularly true in silhouette-based tracking where even visible parts cannot be tracked as long as they do not affect the object silhouette. Multiple cameras or motion priors can overcome this problem. However, multiple cameras or appropriate training data are not always readily available. We propose a framework in which the pose of 3-D models is found by minimising the 2-D projection error through minimisation of an energy function depending on the pose parameters. This framework makes it possible to handle occlusions and self-occlusions by tracking multiple objects and object parts simultaneously. Therefore, each part is described by its own image region each of which is modeled by one probability density function. This allows to deal with occlusions explicitly, which includes self-occlusions between different parts of the same object as well as occlusions between different objects. The results we present for simulations and real-world scenes demonstrate the improvements achieved in monocular and multi-camera settings. These improvements are substantiated by quantitative evaluations, e.g. based on the HumanEVA benchmark.

Real-Time Human Pose Estimation: A Case Study in Algorithm Design for Smart Camera Networks

Monitoring human activities finds novel applications in smart environment settings. Examples include immersive multimedia and virtual reality, smart buildings and occupancy-based services, assisted living and patient monitoring, and interactive classrooms and teleconferencing. A network of cameras can enable detection and interpretation of human events by utilizing multiple views and collaborative processing. Distributed processing of acquired videos at the source camera facilitates operation of scalable vision networks by avoiding transfer of raw images. This allows for efficient collaboration between the cameras under the communication and latency constraints, as well as being motivated by aiming to preserve the privacy of the network users (no image transfer out of the camera) while offering services in applications such as assisted living or virtual placement. In this paper, collaborative processing and data fusion techniques in a multicamera setting are examined in the context of human pose estimation. Multiple mechanisms for information fusion across the space (multiple views), time, and different feature levels are introduced to meet system constraints and are described through examples.