Stereo Camera Research Articles

Fringe photometric stereo

In fringe projection profilometry (FPP), a high spatial frequency of fringe typically indicates powerful error suppression capability; however, it complicates phase unwrapping, necessitating a greater number of patterns and thus compromising the real-time performance of scanning. In this letter, we report a fringe photometric stereo (FPS) to completely bypass phase unwrapping for recovering the continuous 3D surface. We reveal the linear mapping relationship between the phase gradient and depth gradient, thereby facilitating the computation of depth gradients from phase gradients. Thus, we can employ depth gradients to reconstruct high-quality 3D profiles. Experimental results demonstrate that our FPS surpasses traditional phase-shifting profilometry in mitigating Gaussian noise. Our approach enables high-quality and unambiguous 3D reconstruction using single-frequency fringe patterns, relying solely on a camera and a projector without the need for dual, multiple, or light field cameras, thereby paving a path to high-speed and precision 3D imaging.

Visual-Inertial Fusion-Based Five-Degree-of-Freedom Motion Measurement System for Vessel-Mounted Cranes

Vessel-mounted cranes operate in complex marine environments, where precise measurement of cargo positions and attitudes is a key technological challenge to ensure operational stability and safety. This study introduces an integrated measurement system that combines vision and inertial sensing technologies, utilizing a stereo camera and two inertial measurement units (IMUs) to capture cargo motion in five degrees of freedom (DOF). By merging data from the stereo camera and IMUs, the system accurately determines the cargo’s position and attitude relative to the camera. The specific methodology is introduced as follows: First, the YOLO model is adopted to identify targets in the image and generate bounding boxes. Then, using the principle of binocular disparity, the depth within the bounding box is calculated to determine the target’s three-dimensional position in the camera coordinate system. Simultaneously, the IMU measures the attitude of the cargo, and a Kalman filter is applied to fuse the data from the two sensors. Experimental results indicate that the system’s measurement errors in the x, y, and z directions are less than 2.58%, 3.35%, and 3.37%, respectively, while errors in the roll and pitch directions are 3.87% and 5.02%. These results demonstrate that the designed measurement system effectively provides the necessary motion information in 5-DOF for vessel-mounted crane control, offering new approaches for pose detection of marine cranes and cargoes.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence (AI) interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue.
In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

Pixel-level Terrain Processing of the Martian Surface from Moderate Resolution Images of Tianwen-1

The moderate-resolution camera (MoRIC) of the Tianwen-1 orbiter, China’s first Mars exploration mission, is a frame-imaging scheme color camera. However, generating a precise digital elevation model (DEM) from these images faces challenges due to the presence of featureless and repetitive regions on the Martian surface. Traditional photogrammetric methods often struggle to achieve pixel-level resolution in these areas. To address this, we employ a shape from shading (SFS) algorithm that leverages shading patterns to reconstruct fine-scale terrain details, optimizing photogrammetric initial DEM. This paper presents a novel global bisection search algorithm, based on the Pearson correlation coefficient to determine the smoothing parameter, a crucial element in the SFS process. Comparisons with higher-resolution data from the Mars Express high-resolution stereo camera and Mars Reconnaissance Orbiter CTX images validate our approach, highlighting the potential of SFS to significantly enhance the scientific value of Tianwen-1 MoRIC data by generating high-resolution, three-dimensional maps of Mars.

A global colour mosaic of Mars from Mars Express HRSC high altitude observations

The ever-changing transparency of the Martian atmosphere hinders the determination of absolute surface colour from spacecraft images. While individual high-resolution images from low orbit reveal numerous colour details of the geology, the colour variation between images caused by scattering off atmospheric dust can easily be of greater magnitude. The construction of contiguous large-scale mosaics has thus required a strategy to suppress the influence of scattering, often a form of high-pass filtering, which limits their ability to convey colour variation information over distances greater than the dimensions of single images. Here we use a dedicated high altitude observation campaign with the Mars Express High Resolution Stereo Camera (HRSC) (Neukum and Jaumann, 2004; Jaumann et al., 2007), applying a novel iterative method to construct a globally self-consistent colour model. We apply the model to colour-reference a high-altitude mosaic incorporating long-range colour variation information. Using only the relative colour information internal to individual images, the influence of absolute image to image colour changes caused by scattering is minimised, while the model enables colour variations across image boundaries to be self-consistently reconstructed. The resulting mosaic shows a level of colour detail comparable to single images, while maintaining continuity of colour features over much greater distances, thereby increasing the utility of HRSC colour images in the tracing and analysis of martian surface structures.

Pose Estimation of a Cobot Implemented on a Small AI-Powered Computing System and a Stereo Camera for Precision Evaluation.

The precision of robotic manipulators in the industrial or medical field is very important, especially when it comes to repetitive or exhaustive tasks. Geometric deformations are the most common in this field. For this reason, new robotic vision techniques have been proposed, including 3D methods that made it possible to determine the geometric distances between the parts of a robotic manipulator. The aim of this work is to measure the angular position of a robotic arm with six degrees of freedom. For this purpose, a stereo camera and a convolutional neural network algorithm are used to reduce the degradation of precision caused by geometric errors. This method is not intended to replace encoders, but to enhance accuracy by compensating for degradation through an intelligent visual measurement system. The camera is tested and the accuracy is about one millimeter. The implementation of this method leads to better results than traditional and simple neural network methods.

Online extrinsic parameters calibration of on-board stereo cameras based on certifiable optimization

The extrinsic parameters of on-board stereo cameras can be slightly altered due to temperature fluctuations, vibrations, and accidental impacts during automobile driving, leading to significant performance loss in dense stereo matching. In this paper, we propose an online calibration method based on certifiable optimization to address this issue. Initially, sparse feature points are collected based on plane distribution and disparity. A robust optimization model is then developed to minimize the epipolar error, utilizing iterative local optimization to eliminate outliers and determine the 5DOF extrinsic parameters. Subsequently, a relaxation problem is constructed using inliers, and global optimization is performed to certify that the locally optimal results are indeed globally optimal. Comparative experimental results demonstrate that the proposed method offers high accuracy and reliability. Additionally, the quality of the disparity map generated by our calibration method is comparable to that achieved through offline calibration.

High-accuracy fringe projection profilometry without phase unwrapping based on multi-view geometry constraints

The number of fringes and phase unwrapping in fringe projection profilometry result in two key factors. The first is to avoid the problems of excessive fringe patterns, and the second is phase ambiguity. This paper presents a three-dimensional (3D) measurement method without phase unwrapping. This method benefits from the geometric constraints and does not require additional images. Meanwhile, epipolar rectification is performed to calibrate the rotation matrix relationship between the new plane of the dual camera and the plane of the projector. Subsequently, using depth constraints, the point pairs with incorrect 3D positions are effectively eliminated, and the initial parallax map is obtained by establishing epipolar lines of the left and right matching points in the projector domain, obtaining the intersection points, and setting up the threshold for filtering. Finally, a function combining the modulation intensity and phase is proposed to refine the parallax map such that the 3D result is insensitive to phase error. The standard step block and standard ball were used to verify the validity of the proposed method, and the experimental results showed that the root mean square error of the method was 0.052 mm.

Hybrid acoustic modeling: far-field predictions from scanning 3D sound intensity data

This research introduces a novel methodology for predicting the far-field acoustic behavior of complex vibro-acoustic sources, utilizing 3D sound intensity data acquired through Scan&Paint 3D's continuous scanning technology. By combining this experimental dataset with the computational modeling capabilities of the commercial acoustic package Actran, we establish a hybrid method capable of accurately computing noise generated by intricate vibro-acoustic phenomena. The use of a 3D sound intensity probe, in combination with an infrared stereo camera, accelerates the acquisition of acoustic data across an extensive area. This approach not only delivers detailed insights into the source's near-field acoustic behavior but also provides inputs for the computational model, allowing us to model how the vibrating structure radiates. Extensive validation is performed by calculating prediction errors across multiple observation points, ensuring high quality far-field predictions. The integration of extensive experimental data with advanced computational modeling marks a significant leap forward in evaluating the far-field impact of complex acoustic sources. This allows the industrial user to assess the performance of multiple potential mitigation solutions through simulation, quickly, easily, and reliably.

Autonomous Lunar Rover Localization while Fully Scanning a Bounded Obstacle-Rich Workspace.

This article addresses the scanning path plan strategy of a rover team composed of three rovers, such that the team explores unknown dark outer space environments. This research considers a dark outer space, where a rover needs to turn on its light and camera simultaneously to measure a limited space in front of the rover. The rover team is deployed from a symmetric base station, and the rover team's mission is to scan a bounded obstacle-rich workspace, such that there exists no remaining detection hole. In the team, only one rover, the hauler, can locate itself utilizing stereo cameras and Inertial Measurement Unit (IMU). Every other rover follows the hauler, while not locating itself. Since Global Navigation Satellite System (GNSS) is not available in outer space, the localization error of the hauler increases as time goes on. For rover's location estimate fix, one occasionally makes the rover home to the base station, whose shape and global position are known in advance. Once a rover is near the station, it uses its Lidar to measure the relative position of the base station. In this way, the rover fixes its localization error whenever it homes to the base station. In this research, one makes the rover team fully scan a bounded obstacle-rich workspace without detection holes, such that a rover's localization error is bounded by letting the rover home to the base station occasionally. To the best of our knowledge, this article is novel in addressing the scanning path plan strategy, so that a rover team fully scans a bounded obstacle-rich workspace without detection holes, while fixing the accumulated localization error occasionally. The efficacy of the proposed scanning and localization strategy is demonstrated utilizing MATLAB-based simulations.

Dynamic Scene Representation for Docking in Urban Waters Using a Stereo Camera

Abstract To navigate and dock safely in urban waters, an autonomous ferry must assess its safe navigation paths by identifying navigable free space and potential obstacles. A deep understanding of its surroundings is crucial for situational awareness. Knowing which objects are static and dynamic is an indispensable ingredient for path planning algorithms and collision avoidance systems, and is especially useful for docking operations, as moving obstacles can interfere with the optimal path to the dock. To handle all types of obstacles, we present a novel dynamic scene representation for docking in urban waters using a short baseline stereo camera. Obstacles protruding from the water surface are represented with vertically oriented rectangles, known as stixels. The stixels are tracked over consecutive frames with dense optical flow. Together with stereo correspondences, their 3D motion is estimated. Stixels representing the same object are grouped together into line segments which are classified as either dynamic or static based on the motion of its associated stixels. Our dynamic representation presents objects in the 3D camera coordinate system as a bird’s eye view map. Additionally, stereo depth uncertainty can significantly impact depth estimates and is considered using a Kalman filter, providing smooth position and velocity estimates. We test our dynamic scene representation on a real docking scenario recorded with an autonomous ferry using a stereo camera. We evaluate the accuracy of the reconstructed 3D scene points and the velocity estimates of the dynamic objects using LiDAR and GNSS.

Structured-light non-coplanar dual line-scan camera system for complete and accurate point cloud reconstruction in variable motion

Line-scan cameras are promising sensors for in-motion three-dimensional (3-D) reconstruction. For practical industrial applications, it’s necessary to realize complete and accurate reconstruction in a single scan with uncontrollable motion changes. However, line-scan camera systems still have difficulty meeting the demand, even with the favorable side-by-side setup. In this paper, we make changes by slightly separating the viewing planes of the side-by-side coplanar line-scan cameras, and propose a structured-light non-coplanar dual line-scan camera system. Based on the new setup, effective methods are put forward for shape measurement, matching, motion estimation and point cloud stitching, and the proposed system can reconstruct complete and accurate point clouds in the case of evident motion changes. Experimental results prove the performance and advantages of the proposed system. Being independent of strict motion conditions, the proposed system can be applied to various industrial scenes such as inline quality inspection on a conveyor belt, defect detection of a moving vehicle, and tunnel inspection on a mobile carrier.

Middle-Level Fusion YOLO on Multispectral Image to Detect Unhealthy Oil Palm Trees

Abstract Locating the unhealthy oil palm trees is one of the essential things in precision agriculture. In several countries such as Indonesia, Malaysia, and Thailand, Ganoderma disease often attacks oil palm trees. Detection techniques using leaf sample pieces, the possibility of biological changes in the leaf, and the collection process are too tricky. Detection techniques using image samples captured by the drone can be more accessible, but they need to provide complete information related to plant vegetation. This research proposes detecting unhealthy oil palms by capturing top-view images using drone. The drone has dual cameras (RGB and OCN bands) to get more information on plant vegetation. Middle-level fusion YOLO is used to recognize the unhealthy oil palm trees. The data was collected at the Oil Palm Plantation in Bogor, which contains over 100 unhealthy objects. Using multispectral images can improve performance compared to using only RGB images. The proposed method provides better performance than using only RGB images and OCN images with mAP (mean Average Precision) is 0.919. The proposed method provides better performance in detecting unhealthy oil palm trees.

Deformation measurement of aero-engine rotating twisted blade based on multi-vision overlapping area feature registration method

Due to the complexity of the spatial surface of the twisted blade of the aero-engine, there are occlusions and shadows between the blades, resulting in visual blind spots. This causes the loss of certain information in the deformation measurement of twisted blade. A deformation measurement method of rotating twisted blades based on multi-vision overlapping area feature registration is proposed. It aims to solve the problem of limited field of view (FOV) in the deformation measurement of twisted blade, and overcome the problems of small measurement range and difficult measurement of traditional contact sensor method in the rotating state of the blade. First, a calibration and image-matching method based on multi-vision digital image correlation method is proposed. Under the premise of ensuring the simultaneous calibration of multi-vision camera stereo calibration, the matching of image detection points on the twisted blade surface is realized. Second, a three-dimensional reconstruction method based on the feature registration of the overlapping area of the FOV is proposed. It is used to solve the problem that the traditional image stitching method is prone to mismatching and to provide complete point cloud information for the deformation field calculation. Finally, the strain of each point is solved by piecewise point-by-point fitting of the local subdomains of the discrete displacement data, which suppresses the amplification of noise and improves the accuracy of blade strain calculation. In this paper, the deformation measurement experiment is carried out for the rotating twisted blade (up to 6000 rpm). The experimental results show that the proposed method can achieve a displacement measurement accuracy of 5 μm and the strain measurement accuracy of 50 με. This method can provide an important guarantee for the accurate evaluation and safe operation of the key performance parameters of the engine.

A preliminary application and its error estimates of a simple stereo-camera measure system for the far-sea fishery of Pacific saury

Pacific saury is one of the most economically important species in the northwestern Pacific Ocean. The management of this resource relies on precise input of biological data such as body length and is often hindered by a lack of such data on captured fish. This study explores the potential of electronic monitoring (EM) using off-the-shelf stereo cameras to overcome the challenges of collecting and measuring saury body length from the Pacific saury fishery. Using a calibrated WEEVIEW SID WV3000 3D camera, a total of 252 paired images with different shooting angles and distances were obtained for further measurement using Sebastes Stereo Image Analysis Software (SSIAS). The treatments for the measurement distance (MD) were 30 cm, 60 cm, and 100 cm, for the depression angle (DA) were 30°, 60°, and 90°, and for the position angle (PA) were 0°, 45°, and 315° (− 45°). An assistant calibration (AC) in the form of a 16 cm black ruler was also added used. The SSIAS measurement results indicated that the best measurement was obtained with 0° position angle, 90° depression angle, and 30 cm distance from the target fish. The use of AC in the SSIAS + AC measurement was proven to reduce the measurement error from 2.45 to 8.64% to − 1.86 to 0.01%. This study set the baseline for the application of EM on collecting saury body length and the use of AC has been proven to increase the measurement accuracy.

Links between atmospheric aerosols and sea state in the Arctic Ocean

Sea spray emission is the largest mass flux of aerosols to the atmosphere with important impact on atmospheric radiative transfer. However, large uncertainties still exit in constraining this mass flux and its climate forcing, in particular in the Arctic, where sea ice and relatively low wind speed in summer constitute a significantly different regime compared to the global ocean. Sea state conditions and marine boundary layer stability are also critical variables, but their contribution is often overlooked. Here we present concurrent observations of sea state using a novel stereo camera system, of sea spray through coarse mode aerosols, and of meteorological variables to determine boundary layer stability in the Barents and Kara Seas during the 2021 Arctic Century Expedition. Our findings reveal that aerosol concentrations were highest over open waters, closely correlating with wave height, followed by wind speed, wave steepness, and wave age. Notably, these correlations were stronger under unstable marine boundary layer conditions, reflecting immediate sea spray generation. By analysing various combinations of sea and atmospheric variables, we identified the wave height Reynolds number as the most effective indicator of atmospheric sea spray concentration, explaining 57% of its variability in unstable conditions. Our study underscores the need to consider sea state, wind, and boundary layer conditions together to accurately estimate atmospheric sea spray concentrations in the Arctic.

Research on Defect Detection Method of Fusion Reactor Vacuum Chamber Based on Photometric Stereo Vision.

This paper addresses image enhancement and 3D reconstruction techniques for dim scenes inside the vacuum chamber of a nuclear fusion reactor. First, an improved multi-scale Retinex low-light image enhancement algorithm with adaptive weights is designed. It can recover image detail information that is not visible in low-light environments, maintaining image clarity and contrast for easy observation. Second, according to the actual needs of target plate defect detection and 3D reconstruction inside the vacuum chamber, a defect reconstruction algorithm based on photometric stereo vision is proposed. To optimize the position of the light source, a light source illumination profile simulation system is designed in this paper to provide an optimized light array for crack detection inside vacuum chambers without the need for extensive experimental testing. Finally, a robotic platform mounted with a binocular stereo-vision camera is constructed and image enhancement and defect reconstruction experiments are performed separately. The results show that the above method can broaden the gray level of low-illumination images and improve the brightness value and contrast. The maximum depth error is less than 24.0% and the maximum width error is less than 15.3%, which achieves the goal of detecting and reconstructing the defects inside the vacuum chamber.

Comparative analysis of tracking and behavioral patterns between wild-type and genetically modified fruit flies using computer vision and statistical methods

Collective animal behavior occurs in groups and swarms at almost every biological scale, from single-celled organisms to the largest animals on Earth. The intriguing mysteries behind these group behaviors have attracted many scholars, and while it is known that models can reproduce qualitative features of such complex behaviors, this requires data from real animals to demonstrate, and obtaining data on the exact features of these groups is tricky. In this paper, we propose the Hidden Markov Unscented Tracker (HMUT), which combines the state prediction capability of HMM and the high-precision nonlinear processing capability of UKF. This prediction-driven tracking mechanism enables HMUT to quickly adjust tracking strategies when facing sudden changes in target motion direction or rapid changes in speed, reducing the risk of tracking loss. Videos of fruit fly swarm movement in an enclosed environment are captured using stereo cameras. For the captured fruit fly images, the thresholded AKAZE algorithm is first used to detect the positions of individual fruit flies in the images, and the motion of the fruit flies is modeled using a multidimensional hidden Markov model (HMM). Tracking is then performed using the Unscented Kalman Filter algorithm to obtain the flight trajectories of the fruit flies in two camera views. Finally, 3D reconstruction of the trajectories in both views is achieved through polar coordinate constraints, resulting in 3D motion data of the fruit flies. Additionally, the efficiency and accuracy of the proposed algorithm are evaluated by simulating fruit fly swarm movement using the Boids algorithm. Finally, based on the tracked fruit fly flight data, behavioral characteristics of the fruit flies are analyzed from two perspectives. The first is a statistical analysis of the differences between the two behaviors. The second dimension involves clustering trajectory similarity using the DTW method based on fruit fly flight trajectories, further analyzing the similarity within clusters and differences between clusters.

Triple-Camera Rectification for Depth Estimation Sensor.

In this study, we propose a novel rectification method for three cameras using a single image for depth estimation. Stereo rectification serves as a fundamental preprocessing step for disparity estimation in stereoscopic cameras. However, off-the-shelf depth cameras often include an additional RGB camera for creating 3D point clouds. Existing rectification methods only align two cameras, necessitating an additional rectification and remapping process to align the third camera. Moreover, these methods require multiple reference checkerboard images for calibration and aim to minimize alignment errors, but often result in rotated images when there is significant misalignment between two cameras. In contrast, the proposed method simultaneously rectifies three cameras in a single shot without unnecessary rotation. To achieve this, we designed a lab environment with checkerboard settings and obtained multiple sample images from the cameras. The optimization function, designed specifically for rectification in stereo matching, enables the simultaneous alignment of all three cameras while ensuring performance comparable to traditional methods. Experimental results with real camera samples demonstrate the benefits of the proposed method and provide a detailed analysis of unnecessary rotations in the rectified images.

Ultrasonic Levitation as a Handling Tool for In-Space Manufacturing Processes

Abstract 3D printing is one of the key technologies in space exploration. The disparity in gravitational forces between Earth and space presents both challenges and opportunities with regard to material handling. This article examines the potential of employing ultrasonic levitation as a handling tool for substrate-free additive manufacturing processes in microgravity environments. Through preliminary experiments, we demonstrate the feasibility of manipulating polymer powders using acoustic fields while concurrently melting the levitated material. Subsequent experiments conducted in our drop tower facility confirm our ability to manipulate particles with acoustic traps under microgravity conditions. Building upon these findings, we outline plans to further advance our research using an expanded acoustic levitation system capable of three-dimensional object manipulation. Our objectives include moving and orienting large components beyond the wavelength limit in microgravity, manipulating granular raw material while melting it in proximity to the print part, and achieving a semi-continuous fusion of print material with the print part. Therefore, we present an intelligent control strategy based on the results of a digital twin simulation. Furthermore, we utilize a stereo camera combined with computer vision as feedback for the control system to ensure precise handling of the manipulated objects and particles. This study represents a significant advance toward the realization of efficient substrate-free additive manufacturing processes in microgravity environments, with potential applications for in-space manufacturing. Ultimately, this could result in long-term space missions becoming less reliant on supply deliveries, thus reducing cost and additionally enabling faster response to unforeseen issues.