Multi-camera System Research Articles

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

A robust deep networks based multi-object multi-camera tracking system for city scale traffic

YOLOv5 Drone Detection Using Multimodal Data Registered by the Vicon System.

This work is focused on the preliminary stage of the 3D drone tracking challenge, namely the precise detection of drones on images obtained from a synchronized multi-camera system. The YOLOv5 deep network with different input resolutions is trained and tested on the basis of real, multimodal data containing synchronized video sequences and precise motion capture data as a ground truth reference. The bounding boxes are determined based on the 3D position and orientation of an asymmetric cross attached to the top of the tracked object with known translation to the object's center. The arms of the cross are identified by the markers registered by motion capture acquisition. Besides the classical mean average precision (mAP), a measure more adequate in the evaluation of detection performance in 3D tracking is proposed, namely the average distance between the centroids of matched references and detected drones, including false positive and false negative ratios. Moreover, the videos generated in the AirSim simulation platform were taken into account in both the training and testing stages.

A diagnostic room for lower limb amputee based on virtual reality and an intelligent space

This article proposes a virtual reality (VR) system for diagnosing and rehabilitating lower limb amputees. A virtual environment and an intelligent space are the basis of the proposed solution. The target audiences are physiotherapists and doctors, and the aim is to provide a VR-based system to allow visualization and analysis of gait parameters and conformity. The multi-camera system from the intelligent space acquires images from patients during gait. This way, it is possible to generate tridimensional information for the VR-based system. Among the provided functionalities, the user can explore the virtual environment and manage several features, such as gait reproduction and parameters displayed, using a head-mounted display and hand controllers. Besides, the system presents an automatic classifier that can assist physiotherapists and doctors in assessing abnormalities from conventional human gait. We evaluate the system through two quantitative experiments. The first one addresses the performance evaluation of the automatic classifier. The second analysis is through a Likert scale questionnaire submitted to a group of physiotherapists. In this case, the specialists evaluate the existing features of the proposed framework. The results from the questionnaire showed that the virtual environment is suitable for helping track patients’ rehabilitation. Also, the neural network-based classifier results are promising, averaging higher than 91% for all evaluation metrics. Finally, a comparison with related works in the literature highlights the contributions of the proposed solution to the field.

Improved Baited Remote Underwater Video (BRUV) for 24 h Real-Time Monitoring of Pelagic and Demersal Marine Species from the Epipelagic Zone

Bait-based remote underwater video (BRUV) systems are effective devices for remotely observing fish and other marine organisms in challenging environments. The development of a long duration (24 h) surface BRUV observation surveys allowed the monitoring of scarce and elusive pelagic sharks and the direct impact on non-targeted species of longline fishing in the Western Mediterranean. Technological limitations, such as the limited storage capacity and a single surface camera, were improved by (i) adding a deep camera equipped with light (below 80 m depth) and (ii) replacing Gopros with a multi-camera video surveillance system (surface and depth) with a storage capacity of several days and access to real-time observation. Based on a deployment effort of 1884 h video data, we identified 11 blue sharks (Prionace glauca) and one bluntnose sixgill shark (Hexanchus griseus), a deep-sea species that scarcely swims at the surface. The real-time observation capability was a powerful tool for reducing logistical costs and for raising environmental awareness in educational and outreach programmes.

Faster is not always better: Turning performance trade-offs in the inshore squids Doryteuthis pealeii and Illex illecebrosus

Jet-propelled swimmers move differently than fin and flipper-based locomotors, relying heavily on directed, pulsed flows to effect maneuvers and turns. Despite a unique propulsive system involving both fins and a jet, little is currently known about the turning performance of most squids. In this study, the maneuverability of two morphologically and ecologically different squids, the shortfin squid, Illex illecebrosus, and longfin squid, Doryteuthis pealeii, was studied using kinematic tracking approaches. We recorded 128 spontaneous turns using a multi-camera system, tracked several landmarks on the squids, and calculated a range of kinematic parameters, including angular velocity and length-specific turning radius along the yaw axis for all turns. Illex illecebrosus completed faster but broader turns than D. pealeii. Doryteuthis pealeii relied more heavily on its fins for turning, with higher fin flap frequencies and amplitudes than I. illecebrosus. Both species were able to complete tighter turns when oriented arms-first versus tail-first, and I. illecebrosus curled its arms more in the arms-first mode, which likely increased angular velocity through a reduction in moment of inertia. The results of this study indicate that the jet, fins, and arms of squids all play important roles in turning performance, and there is a trade-off in squids between maximizing angular velocity and minimizing turning radius. SummaryA kinematic analysis of maneuverability and agility of two inshore squid species: Doryteuthis pealeii and Illex illecebrosus.

METRIC—Multi-Eye to Robot Indoor Calibration Dataset

Multi-camera systems are an effective solution for perceiving large areas or complex scenarios with many occlusions. In such a setup, an accurate camera network calibration is crucial in order to localize scene elements with respect to a single reference frame shared by all the viewpoints of the network. This is particularly important in applications such as object detection and people tracking. Multi-camera calibration is a critical requirement also in several robotics scenarios, particularly those involving a robotic workcell equipped with a manipulator surrounded by multiple sensors. Within this scenario, the robot-world hand-eye calibration is an additional crucial element for determining the exact position of each camera with respect to the robot, in order to provide information about the surrounding workspace directly to the manipulator. Despite the importance of the calibration process in the two scenarios outlined above, namely (i) a camera network, and (ii) a camera network with a robot, there is a lack of standard datasets available in the literature to evaluate and compare calibration methods. Moreover they are usually treated separately and tested on dedicated setups. In this paper, we propose a general standard dataset acquired in a robotic workcell where calibration methods can be evaluated in two use cases: camera network calibration and robot-world hand-eye calibration. The Multi-Eye To Robot Indoor Calibration (METRIC) dataset consists of over 10,000 synthetic and real images of ChAruCo and checkerboard patterns, each one rigidly attached to the robot end-effector, which was moved in front of four cameras surrounding the manipulator from different viewpoints during the image acquisition. The real images in the dataset includes several multi-view image sets captured by three different types of sensor networks: Microsoft Kinect V2, Intel RealSense Depth D455 and Intel RealSense Lidar L515, to evaluate their advantages and disadvantages for calibration. Furthermore, in order to accurately analyze the effect of camera-robot distance on calibration, we acquired a comprehensive synthetic dataset, with related ground truth, with three different camera network setups corresponding to three levels of calibration difficulty depending on the cell size. An additional contribution of this work is to provide a comprehensive evaluation of state-of-the-art calibration methods using our dataset, highlighting their strengths and weaknesses, in order to outline two benchmarks for the two aforementioned use cases.

COLMAP-SLAM: A FRAMEWORK FOR VISUAL ODOMETRY

Abstract. SLAM technology is more and more integrated with other sensors for indoor and outdoor seamless navigation. This research topic is very active in particular on image matching with deep learning local features, keyframe selection approaches, or tests on new IMU and GNSS solutions. Integrating and testing new methodologies on other widely used SLAM implementations, such as ORB-SLAM, can be not a trivial task. Therefore, we propose an extension of COLMAP to be used in real-time as a feature-based Visual-SLAM that can be also coupled with other sensors. COLMAP has been chosen due to its modularity and the large community that assures the continuity of the repository. The paper presents a pipeline mainly thought for real-time evaluation of learning-based tie points and new SLAM features, that works with both monocular, stereo and multi-camera systems. It is also shown an example of keyframe selection algorithm based on deep learning local features, and a simple example of IMU integration. The code is available on the GitHub repository https://github.com/3DOM-FBK/COLMAP_SLAM.

Conducting an Analysis of Mosquito Flight Behaviors in a Wind Tunnel.

Prior to conducting wind tunnel experiments, mosquitoes must be prepared for testing. Important factors and state-dependent processes of the mosquito-like the sex, age, infection status, reproductive status, or nutritional status-should be evaluated and motivated by questions and hypotheses one seeks to address. Other critical external factors that can impact the mosquitoes' behavior and should be controlled for both in the colony and in the room where the wind tunnel experiments take place include the circadian rhythm, room temperature, light intensity, and relative humidity. Together, the internal and external factors, and wind tunnel design, ultimately control the behavior of the mosquito and, hence, the success of the experiments. In the present protocol, we describe methods using a standard wind tunnel design in which the fan pulls the air through the working section of the wind tunnel and the mosquito behavior is recorded by a multicamera system. Variations around the camera tracking system can be adapted according to the research questions being asked and include real-time tracking for both closed-loop and open-loop control of the stimulus environment or recording of the videos for off-line digitization and analysis. Within the working section, the sensory environment (odor, visual, wind) can be controlled to test the mosquito responses to different stimuli, and below we include different equipment and tools for modifying the stimuli the mosquito experiences during flight. Finally, the methods described here are applicable to multiple mosquito species, although the experiment parameters may need to be changed (e.g., ambient luminosity).

The generation mechanism and identification of spatially non-continuous error in integral multi-station measurement systems

The integral multi-station measurement system represented by indoor GPS, workshop Measuring and Positioning System or a multi-camera system, realizes high-precision 3D measurement in large-scale space by constructing a global measurement field within a unified temporal-spatial reference. In advanced equipment manufacturing sites such as robot machining, large component (the fuselage or wing of airplanes) assembly and monitoring, the integral multi-station measurement system overcomes the drawbacks of the single-station measurement system such as laser tracker that can hardly balance the measurement range and accuracy. However, with the improvement of automation and intelligence degree in equipment manufacturing, the requirements on accuracy and robustness has become more and more strict, which is a huge challenge to existing measuring technologies. Limited by the measurement principle of multi-observation intersection, the change of geometry constraint in the integral multi-station measurement field will lead to spatially non-continuous error, degrading the performance of the integral multi-station measurement system. In this paper we first analyze the mechanisms and characteristics of spatially non-continuous error at the geometry constraint level. Basing on the error propagation model, a criterion for evaluating the significance of non-continuous error is then proposed for error identification. Finally, simulations and experiments are carried out, and this proposed criterion is verified to be effective.

A REPLICABLE OPEN-SOURCE MULTI-CAMERA SYSTEM FOR LOW-COST 4D GLACIER MONITORING

Abstract. Image-based monitoring has emerged as a prevalent technique for sensing mountain environments. Monoscopic time-lapse cameras, which rely on digital image correlation to quantify glacier motion, have limitations due to the need for a Digital Elevation Model for deriving 3D flow velocity fields. Multi-camera systems overcome this limitation, as they allow for a 3D reconstruction of the scene. This paper presents a replicable low-cost stereoscopic system designed for 4D glacier monitoring. The system consists of independent and autonomous units, built from off-the-shelves components, such as a DSLR camera, an Arduino microcontroller, and a Raspberry Pi Zero, reducing costs compared to pre-built time-lapse cameras. The units are energetically self-sufficient and resistant to harsh alpine conditions. The system was successfully tested for more than a year to monitor the northwest terminus of the Belvedere Glacier (Italian Alps). Daily stereo-pairs acquired were processed with Structure-from-Motion to derive 3D point clouds of the glacier terminus and estimate glacier retreat and ice volume loss. By combining the information about ice volume loss with ablation estimates and ice flow velocity information, e.g., derived from monoscopic-camera time series, a multi-camera system enables a comprehensive understanding of sub-seasonal glacier dynamics.

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

A robust attribute-aware and real-time multi-target multi-camera tracking system using multi-scale enriched features and hierarchical clustering

A new method for localized necking detection in uniaxial tensile testing based on a multi-camera DIC system

Understanding and predicting the onset and behavior of localized necking is important for the design and optimization of structures and components. Capturing the onset of localized necking more accurately has been a challenge in experimental mechanics. In this paper, a new alternative method for detecting the onset of localized necking in uniaxial tensile tests is presented. This method is inspired by the fact that different behaviors are exhibited as localized necking starts at the different regions of the cross-sectional area. When localized necking begins to occur, the cross-sectional area in the center of the necking area decreases sharply, while the cross-sectional area at the boundary of the necking area no longer decreases. A multi-camera digital image correlation system consisting of four cameras, two placed at the front and two at the back of the specimen, has been developed to measure the cross-sectional area at different positions of the sample. The effectiveness and accuracy of the proposed method are verified through a series of tensile tests and are compared with different localized necking detection methods.

Detection of Empty/Occupied States of Parking Slots in Multicamera system using Mask R-CNN Classifier

A fast growth of vehicles in big cities has an impact of arising road loads and difficulty of finding empty parking spaces. One solution to cope with the problem is to develop a parking management system which can provide useful information of available parking spaces to the potential users. This paper discusses about a new multicamera arrangement and the function to evaluate the empty/occupied states of the parking slots, as an alternative solution to the existing single camera system, The system adopted Mask R-CNN for its classifier, because of its capability to provide the polygon outputs for its detected objects, compared with the existing bounding box outputs provided by other classifiers. The proposed function has optimized the available information from all cameras, by considering the relative position of each camera to the parking spaces, and also capable of overcoming occlusion problem occurs in some cameras, The experiment shows that the capability of overcoming the occlusion problem has been validated, and its performance to evaluate the empty/occupied states of the parking slots was better than the single camera system to a certain threshold.

Developing digital twins of multi-camera metrology systems in Blender

Blender is an open-source three-dimensional animation software, which can be used as a simulation tool in metrology, to build numerical models that can be used in the design and optimisation of camera-based measurement systems. In this work, the relevance of using Blender to model camera-based measurement systems was explored. Two experiments were conducted in real-world and Blender modelled environments, one using individual cameras for a simple measurement task, the other considering multi-camera position optimisation. The objective was to verify whether the virtual cameras created in Blender can perceive and measure objects in the same manner as the real cameras in an equivalent environment. The results demonstrate that in its native modelling format Blender satisfies the optical metrology characteristics of measurement, but the correlation between Blender output and real-world results is highly sensitive to initial modelling parameters such as illumination intensity, camera definitions and object surface texture.

Roving Multiple Camera Array with Structure-from-Motion for Coastal Monitoring

Regular monitoring is essential for vulnerable coastal locations such as areas of landward retreat. However, for coastal practitioners, surveying is limited by budget, specialist personnel/equipment and weather. In combination structure-from-motion and multi-view stereo (SfM-MVS) has helped to improve accessibility to topographic data acquisition. Pole-mounted cameras with SfM-MVS have gained traction but to guarantee coverage and reconstruction quality, greater understanding of camera position and interaction is required. This study uses a multi-camera array for image acquisition and reviews processing procedures in Agisoft Photoscan (Metashape). The camera rig was deployed at three sites and results were verified against a terrestrial laser scanner (TLS) and independent precision estimates. The multi-camera approach provided effective image acquisition ~11 times faster than the TLS. Reconstruction quality equalled (>92% similarity) the TLS, subject to processing parameters. A change in the image alignment parameter demonstrated a significant influence on deformation, reducing reprojection error by~94%. A lower densification parameter (‘High’) offered results ~4.39% dissimilar from the TLS at 1/8th of the processing time of other parameters. Independent precision estimates were <8.2 mm for x, y and z dimensions. These findings illustrate the potential of multi-camera systems and the influence of processing on point cloud quality and computation time.

Multi-camera BEV video-surveillance system for efficient monitoring of social distancing

The current sanitary emergency situation caused by COVID-19 has increased the interest in controlling the flow of people in indoor infrastructures, to ensure compliance with the established security measures. Top view camera-based solutions have proven to be an effective and non-invasive approach to accomplish this task. Nevertheless, current solutions suffer from scalability problems: they cover limited range areas to avoid dealing with occlusions and only work with single camera scenarios. To overcome these problems, we present an efficient and scalable people flow monitoring system that relies on three main pillars: an optimized top view human detection neural network based on YOLO-V4, capable of working with data from cameras at different heights; a multi-camera 3D detection projection and fusion procedure, which uses the camera calibration parameters for an accurate real-world positioning; and a tracking algorithm which jointly processes the 3D detections coming from all the cameras, allowing the traceability of individuals across the entire infrastructure. The conducted experiments show that the proposed system generates robust performance indicators and that it is suitable for real-time applications to control sanitary measures in large infrastructures. Furthermore, the proposed projection approach achieves an average positioning error below 0.2 meters, with an improvement of more than 4 times compared to other methods.

Toward Multicamera Systems for Minimally Invasive Surgery

Nowadays, many surgical operations for the diagnosis and treatment of diseases are performed using minimally invasive techniques. This trend has increased the demand for ergonomic medical devices to help surgeons safely execute medical practices by enhancing their experience, thus improving surgical outcomes. In this context, one of the most important challenges of minimally invasive surgery is the narrow field of view: Surgeons can only see the instrument's tip movement on a 2-D screen and cannot monitor the adjacent anatomical areas in three dimensions as in traditional open surgery. Hence, perforation of healthy tissues or undesired brushing may occur. To cope with these problems, we propose a novel surgical instrument, which embeds multiple fisheye cameras, accompanied by an image processing algorithm to visualize the adjacent area from different angles. This will allow surgeons to have a broader view around the instrument itself, hence preventing injuries caused by contact between the instrument body and the surrounding anatomical areas. The proposed system is versatile: It can be used for different surgical operations, and it can also be adapted for other applications dealing with narrow space exploration requiring a broad view.

A Hardware and Software Platform for Aerial Object Localization

To date, there are little reliable data on the position, velocity and acceleration characteristics of Unidentified Aerial Phenomena (UAP). The dual hardware and software system described in this document provides a means to address this gap. We describe a weatherized multi-camera system which can capture images in the visible, infrared and near infrared wavelengths. We then describe the software we will use to calibrate the cameras and to robustly localize objects-of-interest in three dimensions. We show how object localizations captured over time will be used to compute the velocity and acceleration of airborne objects.

Full-field modal identification using reliability-guided frequency-domain-based digital image correlation method based on multi-camera system

A multi-camera array system was established to address the limitations of traditional measurement methods in full-field modal measurement. According to the principle of binocular vision, two cameras are utilized to form a set of three-dimensional measurement subsystems, and the subsystems are calibrated to establish unity in coordinates. To improve computational efficiency, a reliability-guided frequency-domain-based digital image correlation method is proposed to optimize pixel matching and calculation path. This method incorporates the benefits of a reliability-guided strategy and an integer-pixel search method based on the frequency domain to ensure that the seed point with the optimal matching effect in the calculation area is selected. The zero-mean normalized cross-correlation coefficient is used as the criterion for determining reliability. This approach effectively avoids the problem of error transmission and reduces the need for complex operations and repeated searches in the spatial domain, thus improving computational efficiency. The matching time is reduced, the calculation efficiency of displacement signal extraction under large data volume is improved, and the obtained response signal is substituted into the modal identification algorithm to obtain the modal parameters of the structure. Ultimately, the modal test of a solar panel is performed by constructing a four-camera system, which verifies the correctness of the proposed method and has practical engineering significance.

Accuracy and repeatability of the Microsoft Azure Kinect for clinical measurement of motor function.

Quantitative assessment of motor function is increasingly applied in fall risk stratification, diagnosis, and disease monitoring of neuro-geriatric disorders of balance and gait. Its broad application, however, demands for low-cost and easy to use solutions that facilitate high-quality assessment outside laboratory settings. In this study, we validated in 30 healthy adults (12 female, age: 32.5 [22 - 62] years) the performance and accuracy of the latest generation of the Microsoft RGB-D camera, i.e., Azure Kinect (AK), in tracking body motion and providing estimates of clinical measures that characterise static posture, postural transitions, and locomotor function. The accuracy and repeatability of AK recordings was validated with a clinical reference standard multi-camera motion capture system (Qualisys) and compared to its predecessor Kinect version 2 (K2). Motion signal quality was evaluated by Pearson's correlation and signal-to-noise ratios while the accuracy of estimated clinical parameters was described by absolute and relative agreement based on intraclass correlation coefficients. The accuracy of AK-based body motion signals was moderate to excellent (RMSE 89 to 20 mm) and depended on the dimension of motion (highest for anterior-posterior dimension), the body region (highest for wrists and elbows, lowest for ankles and feet), and the specific motor task (highest for stand up and sit down, lowest for quiet standing). Most derived clinical parameters showed good to excellent accuracy (r .84 to .99) and repeatability (ICC(1,1) .55 to .94). The overall performance and limitations of body tracking by AK were comparable to its predecessor K2 in a cohort of young healthy adults. The observed accuracy and repeatability of AK-based evaluation of motor function indicate the potential for a broad application of high-quality and long-term monitoring of balance and gait in different non-specialised environments such as medical practices, nursing homes or community centres.