Multiple Camera Setups Research Articles

MEDUSA: Marine benthic Ecological Data from Underwater imagery Surveys of sub-Antarctic Crozet environments

Inscribed on the UNESCO World Heritage list, the sub-Antarctic Crozet archipelago is located in a region facing significant environmental changes impacting a poorly known marine biodiversity. Underwater imagery constitutes a valuable non-invasive approach for gathering ecological data and improving our knowledge of ecosystems’ vulnerability. We here compiled two datasets, encompassing 17 video-imagery surveys of Crozet nearshore environments conducted in 2021 and 2022 at two sites of Ile de la Possession: Baie du Marin and Crique du Sphinx. Faunal abundance and algal cover data related to each survey are also provided. A total of 755 images were analysed, comprising 52 faunal and 14 algal taxa identified in 2021, as well as 45 faunal and 14 algal taxa identified in 2022. Video-transects were performed in shallow waters by scuba divers using a GoPro®HERO7 multiple camera set-up, and in deeper waters using a remotely operated vehicle. These data provide a first baseline for biodiversity and ecosystem studies, and for monitoring the long-term dynamics of Crozet benthic habitats facing natural and anthropogenic disturbances.

3D Scene Geometry Estimation from 360° Imagery: A Survey

This article provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under omnidirectional optics. We first revisit the basic concepts of the spherical camera model and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360°, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this article by pointing out current and future trends.

Spatio-Temporal Calibration of Multiple Kinect Cameras Using 3D Human Pose.

RGB and depth cameras are extensively used for the 3D tracking of human pose and motion. Typically, these cameras calculate a set of 3D points representing the human body as a skeletal structure. The tracking capabilities of a single camera are often affected by noise and inaccuracies due to occluded body parts. Multiple-camera setups offer a solution to maximize coverage of the captured human body and to minimize occlusions. According to best practices, fusing information across multiple cameras typically requires spatio-temporal calibration. First, the cameras must synchronize their internal clocks. This is typically performed by physically connecting the cameras to each other using an external device or cable. Second, the pose of each camera relative to the other cameras must be calculated (Extrinsic Calibration). The state-of-the-art methods use specialized calibration session and devices such as a checkerboard to perform calibration. In this paper, we introduce an approach to the spatio-temporal calibration of multiple cameras which is designed to run on-the-fly without specialized devices or equipment requiring only the motion of the human body in the scene. As an example, the system is implemented and evaluated using Microsoft Azure Kinect. The study shows that the accuracy and robustness of this approach is on par with the state-of-the-art practices.

Fast, accurate and robust registration of multiple depth sensors without need for RGB and IR images

Registration is an essential prerequisite for many applications when a multiple-camera setup is used. Due to the noise in depth images, registration procedures for depth sensors frequently rely on the detection of a target object in color or infrared images. However, this prohibits use cases where color and infrared images are not available or where there is no mapping between the pixels of different image types, e.g., due to separate sensors or different projections. We present our novel registration method that requires only the point cloud resulting from the depth image of each camera. For feature detection, we propose a combination of a custom-designed 3D registration target and an algorithm that is able to reliably detect that target and its features in noisy point clouds. Our evaluation indicates that our lattice detection is very robust (with a precision of more than 0.99) and very fast (on average about 20 ms with a single core). We have also compared our registration method with known methods: Our registration method achieves an accuracy of 1.6 mm at a distance of 2 m using only the noisy depth image, while the most accurate registration method achieves an accuracy of 0.7 mm requiring both the infrared and depth image.

Three-Dimensional Tracking of Multiple Small Insects by a Single Camera.

Many systems to monitor insect behavior have been developed recently. Yet most of these can only detect two-dimensional behavior for convenient analysis and exclude other activities, such as jumping or flying. Therefore, the development of a three-dimensional (3D) monitoring system is necessary to investigate the 3D behavior of insects. In such a system, multiple-camera setups are often used to accomplish this purpose. Here, a system with a single camera for tracking small insects in a 3D space is proposed, eliminating the synchronization problems that typically occur when multiple cameras are instead used. With this setup, two other images are obtained via mirrors fixed at other viewing angles. Using the proposed algorithms, the tracking accuracy of five individual drain flies, Clogmia albipunctata (Williston) (Diptera: Psychodidae), flitting about in a spherical arena (78 mm in diameter) is as high as 98.7%, whereas the accuracy of 10 individuals is 96.3%. With this proposed method, the 3D trajectory monitoring experiments of insects can be performed more efficiently.

EOMVS: Event-Based Omnidirectional Multi-View Stereo

Event-based cameras sense changes in light intensity, and asynchronously outputs event data. Compared with conventional cameras, event-based cameras have the advantages of low latency, high dynamic range, and low power consumption. Event cameras have been utilized for various vision tasks, such as depth estimation and object detection under severe illumination changes and dynamic motion. Fisheye or omnidirectional cameras, on the other hand, have much wider field-of-view (FoV) allowing a more compact system for omnidirectional vision in comparison with multiple normal-FoV camera setups. In this work, we propose a new multi-view stereo method, called EOMVS, to reconstruct a 3D scene with a wide view using the event data captured by omnidirectional event cameras. The proposed method has the advantages of both event and fisheye cameras. To validate our EOMVS method, we conducted experiments using both synthetic and real-world datasets, and evaluated the performance both qualitatively and quantitatively. The experiments show that the presented approach can accurately obtain wide-view 3D information.

Analysis of Pedestrian Hybrid Beacon Operation on Higher-Speed Roadways

The pedestrian hybrid beacon (PHB) is a traffic control device used at pedestrian crossings. It was first included in the 2009 Manual on Uniform Traffic Control Devices and was based on the HAWK developed in Tucson, Arizona. The focus of an Arizona Department of Transportation research study was the investigation of the use of PHBs on higher-speed roads. Ten locations in Arizona representing higher operating speed conditions (85th percentile speed ranging from 44 to 54 mph) were selected for inclusion in this study. Data were collected using a multiple video camera setup. The final dataset reflected about 40 h of video data and included 1,214 pedestrians or cyclists crossing at PHBs. Overall, driver yielding for these 10 sites averaged 97%, which is similar to driver-yielding rates for PHBs installed on lower-speed streets.

Modeling and vision-based control of large-dimension cable-driven parallel robots using a multiple-camera setup

This paper deals with the modeling and vision-based control of large-dimension cable-driven parallel robots. Inverse kinematics and instantaneous inverse kinematics models are derived from the elastic catenary cable modeling. These models turn out to be dependent on the pose of the mobile platform (end-effector), on the cable tangent directions and on the cable tensions. In order to control the motion of the robot, a position-based visual servo control is used, where the mobile platform pose is measured by vision and used for regulation. A multi-camera setup and load cells provide the aforementioned desired measurements, i.e., the mobile platform pose, the directions of the tangents to the cables, and the cable tensions. The proposed approach was validated in experiments on the large-dimension cable-driven parallel robot prototype CoGiRo of global dimensions 15 m × 11 m × 6 m (L × l × h). A maximum error of less than 1 cm in position and 0.5∘ in orientation was achieved. Moreover, in the case of cable-driven parallel robots larger than the prototype CoGiRo, simulations were conducted in order to assess the influence on the vision-based control of four instantaneous inverse kinematics models.

Appearance capture and modeling of human teeth

Recreating the appearance of humans in virtual environments for the purpose of movie, video game, or other types of production involves the acquisition of a geometric representation of the human body and its scattering parameters which express the interaction between the geometry and light propagated throughout the scene. Teeth appearance is defined not only by the light and surface interaction, but also by its internal geometry and the intra-oral environment, posing its own unique set of challenges. Therefore, we present a system specifically designed for capturing the optical properties of live human teeth such that they can be realistically re-rendered in computer graphics. We acquire our data in vivo in a conventional multiple camera and light source setup and use exact geometry segmented from intra-oral scans. To simulate the complex interaction of light in the oral cavity during inverse rendering we employ a novel pipeline based on derivative path tracing with respect to both optical properties and geometry of the inner dentin surface. The resulting estimates of the global derivatives are used to extract parameters in a joint numerical optimization. The final appearance faithfully recreates the acquired data and can be directly used in conventional path tracing frameworks for rendering virtual humans.

Marker-free monitoring of the grandstand structures and modal identification using computer vision methods

In this study, a vision-based multi-point structural dynamic monitoring framework is proposed. This framework aims to solve issues in current vision-based structural health monitoring. Limitations are due to manual markers, single-point monitoring, and synchronization between a multiple-camera setup and a sensor network. The proposed method addresses the first issue using virtual markers—features extracted from an image—instead of physical manual markers. The virtual markers can be selected according to each scenario, which makes them versatile. The framework also overcomes the issue of single-point monitoring by utilizing an advanced visual tracking algorithm based on optical flow, allowing multi-point displacement measurements. Besides, a synchronization mechanism between a multiple-camera setup and a sensor network is built. The proposed method is first tested on a grandstand simulator located in the laboratory. The experiment is to verify the performance of displacement measurement of the proposed method and conduct structural identification of the grandstand through multi-point displacement records. The results from the proposed method are then compared to the data gathered by traditional displacement sensors and accelerometers. A second experiment is conducted at a stadium during a football game to validate the feasibility of field application and the operational modal identification of the stadium under human crowd jumping through the measured displacement records. From these experiments, it is concluded that the proposed method can be employed to identify modal parameters for structural health monitoring.

Fisher Motion Descriptor for Multiview Gait Recognition

The goal of this paper is to identify individuals by analyzing their gait. Instead of using binary silhouettes as input data (as done in many previous works) we propose and evaluate the use of motion descriptors based on densely sampled short-term trajectories. We take advantage of state-of-the-art people detectors to define custom spatial configurations of the descriptors around the target person, obtaining a rich representation of the gait motion. The local motion features (described by the Divergence-Curl-Shear descriptor [M. Jain, H. Jegou and P. Bouthemy, Better exploiting motion for better action recognition, in Proc. IEEE Conf. Computer Vision Pattern Recognition (CVPR) (2013), pp. 2555–2562.]) extracted on the different spatial areas of the person are combined into a single high-level gait descriptor by using the Fisher Vector encoding [F. Perronnin, J. Sánchez and T. Mensink, Improving the Fisher kernel for large-scale image classification, in Proc. European Conf. Computer Vision (ECCV) (2010), pp. 143–156]. The proposed approach, coined Pyramidal Fisher Motion, is experimentally validated on ‘CASIA’ dataset [S. Yu, D. Tan and T. Tan, A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition, in Proc. Int. Conf. Pattern Recognition, Vol. 4 (2006), pp. 441–444]. (parts B and C), ‘TUM GAID’ dataset, [M. Hofmann, J. Geiger, S. Bachmann, B. Schuller and G. Rigoll, The TUM Gait from Audio, Image and Depth (GAID) database: Multimodal recognition of subjects and traits, J. Vis. Commun. Image Represent. 25(1) (2014) 195–206]. ‘CMU MoBo’ dataset [R. Gross and J. Shi, The CMU Motion of Body (MoBo) database, Technical Report CMU-RI-TR-01-18, Robotics Institute (2001)]. and the recent ‘AVA Multiview Gait’ dataset [D. López-Fernández, F. Madrid-Cuevas, A. Carmona-Poyato, M. Marín-Jiménez and R. Muñoz-Salinas, The AVA multi-view dataset for gait recognition, in Activity Monitoring by Multiple Distributed Sensing, Lecture Notes in Computer Science (Springer, 2014), pp. 26–39]. The results show that this new approach achieves state-of-the-art results in the problem of gait recognition, allowing to recognize walking people from diverse viewpoints on single and multiple camera setups, wearing different clothes, carrying bags, walking at diverse speeds and not limited to straight walking paths.

Volume Estimation of Mango

This paper observes the possibility of volume estimate of mango using image processing technique that are used in commercial packaging industry for increase the quality of fruits. Image processing is used to provide automation and cost-effective service to packaging industry for volume estimation of mango. Sizes featured are extracted from digit profile image using many image processing technique. The volume of fruit is obtained using single or multiple camera setups. The sizes featured are used to estimated volume of mango and compare with actual volume of mango. That program can measure the size property of mango such as height, width and area from image but there is one problem is that the size feature of mango is not accurate. We have to done some changes to achieve high accuracy of size featured of mango.

REAL-TIME OBJECT DETECTION, TRACKING, AND 3D POSITIONING IN A MULTIPLE CAMERA SETUP

Abstract. We introduce a real-time object positioning system that naturally combines detection, tracking, and 3D positioning in a multiple camera setup. Depending on the task, the detection, tracking, and positioning steps, when performed independently have limitations due to the loss of temporal and geometric constraints. We conjecture that the detection and tracking steps do not always estimate a correct solution due to incorrect matches, appearance variations or occlusions. In order to offset these problems, we use the geometry between the target object and cameras to refine detection and tracking result and provide persistent and reliable solution to 3D positioning. The geometry is introduced to the detection and tracking steps by simultaneously solving the 3D position of the target object by using a bundle adjustment step. Test results show that the system provides reliable solution at the speed of six to ten frames per second for a four high definition camera setup running on a single computer.

Skeleton body pose tracking from efficient three-dimensional motion estimation and volumetric reconstruction

We address the problem of body pose tracking in a scenario of multiple camera setup with the aim of recovering body motion robustly and accurately. The tracking is performed on three-dimensional (3D) space using 3D data, including colored volume and 3D optical flow, which are reconstructed at each time step. We introduce strategies to compute multiple camera-based 3D optical flow and have attained efficient and robust 3D motion estimation. Body pose estimation starts with a prediction using 3D optical flow and then is changed to a lower-dimensional global optimization problem. Our method utilizes a voxel subject-specific body model, exploits multiple 3D image cues, and incorporates physical constraints into a stochastic particle-based search initialized from the deterministic prediction and stochastic sampling. It leads to a robust 3D pose tracker. Experiments on publicly available sequences show the robustness and accuracy of our approach.

Hardware Middleware for Person Tracking on Embedded Distributed Smart Cameras

Tracking individuals is a prominent application in such domains like surveillance or smart environments. This paper provides a development of a multiple camera setup with jointed view that observes moving persons in a site. It focuses on a geometry-based approach to establish correspondence among different views. The expensive computational parts of the tracker are hardware accelerated via a novel system-on-chip (SoC) design. In conjunction with this vision application, a hardware object request broker (ORB) middleware is presented as the underlying communication system. The hardware ORB provides a hardware/software architecture to achieve real-time intercommunication among multiple smart cameras. Via a probing mechanism, a performance analysis is performed to measure network latencies, that is, time traversing the TCP/IP stack, in both software and hardware ORB approaches on the same smart camera platform. The empirical results show that using the proposed hardware ORB as client and server in separate smart camera nodes will considerably reduce the network latency up to 100 times compared to the software ORB.

Silhouette Segmentation in Multiple Views

In this paper, we present a method for extracting consistent foreground regions when multiple views of a scene are available. We propose a framework that automatically identifies such regions in images under the assumption that, in each image, background and foreground regions present different color properties. To achieve this task, monocular color information is not sufficient and we exploit the spatial consistency constraint that several image projections of the same space region must satisfy. Combining the monocular color consistency constraint with multiview spatial constraints allows us to automatically and simultaneously segment the foreground and background regions in multiview images. In contrast to standard background subtraction methods, the proposed approach does not require a priori knowledge of the background nor user interaction. Experimental results under realistic scenarios demonstrate the effectiveness of the method for multiple camera set ups.

Object Plane Deformation Due to Refraction in Two-Dimensional Underwater Motion Analysis

The purpose of this study was twofold: (a) to investigate the effects of selected experimental factors on the magnitude of the object plane deformation due to refraction, and (b) to discuss their practical implications in an effort to improve the applicability of the 2-D DLT method in the underwater motion analysis. The RMS and maximum object plane reconstruction errors of various experimental conditions were computed systematically. To isolate the error due to refraction from the experimental errors, the comparator coordinates (image plane coordinates) of the control points were computed based on a theoretical refraction model rather than actual digitizing. It was concluded from a series of object plane reconstruction that among the distance and angle factors of the experimental setting in the 2-D underwater motion analysis, the camera-to-interface distance and the interface-to-control-object distance are the two major factors affecting the magnitude of the object plane deformation. The other factors revealed only minor effects. The advantages of the 2-D DLT method over the traditional multiplier method in underwater motion analysis. such as oblique projection and multiple camera setup. were further discussed. Possible ways to reduce the maximum reconstruction error were also explored.