Stereo Camera Research Articles

Advanced Monocular Outdoor Pose Estimation in Autonomous Systems: Leveraging Optical Flow, Depth Estimation, and Semantic Segmentation with Dynamic Object Removal

Autonomous technologies have revolutionized transportation, military operations, and space exploration, necessitating precise localization in environments where traditional GPS-based systems are unreliable or unavailable. While widespread for outdoor localization, GPS systems face limitations in obstructed environments such as dense urban areas, forests, and indoor spaces. Moreover, GPS reliance introduces vulnerabilities to signal disruptions, which can lead to significant operational failures. Hence, developing alternative localization techniques that do not depend on external signals is essential, showing a critical need for robust, GPS-independent localization solutions adaptable to different applications, ranging from Earth-based autonomous vehicles to robotic missions on Mars. This paper addresses these challenges using Visual odometry (VO) to estimate a camera’s pose by analyzing captured image sequences in GPS-denied areas tailored for autonomous vehicles (AVs), where safety and real-time decision-making are paramount. Extensive research has been dedicated to pose estimation using LiDAR or stereo cameras, which, despite their accuracy, are constrained by weight, cost, and complexity. In contrast, monocular vision is practical and cost-effective, making it a popular choice for drones, cars, and autonomous vehicles. However, robust and reliable monocular pose estimation models remain underexplored. This research aims to fill this gap by developing a novel adaptive framework for outdoor pose estimation and safe navigation using enhanced visual odometry systems with monocular cameras, especially for applications where deploying additional sensors is not feasible due to cost or physical constraints. This framework is designed to be adaptable across different vehicles and platforms, ensuring accurate and reliable pose estimation. We integrate advanced control theory to provide safety guarantees for motion control, ensuring that the AV can react safely to the imminent hazards and unknown trajectories of nearby traffic agents. The focus is on creating an AI-driven model(s) that meets the performance standards of multi-sensor systems while leveraging the inherent advantages of monocular vision. This research uses state-of-the-art machine learning techniques to advance visual odometry’s technical capabilities and ensure its adaptability across different platforms, cameras, and environments. By merging cutting-edge visual odometry techniques with robust control theory, our approach enhances both the safety and performance of AVs in complex traffic situations, directly addressing the challenge of safe and adaptive navigation. Experimental results on the KITTI odometry dataset demonstrate a significant improvement in pose estimation accuracy, offering a cost-effective and robust solution for real-world applications.

Deep Neural Networks for Accurate Depth Estimation with Latent Space Features

Depth estimation plays a pivotal role in advancing human–robot interactions, especially in indoor environments where accurate 3D scene reconstruction is essential for tasks like navigation and object handling. Monocular depth estimation, which relies on a single RGB camera, offers a more affordable solution compared to traditional methods that use stereo cameras or LiDAR. However, despite recent progress, many monocular approaches struggle with accurately defining depth boundaries, leading to less precise reconstructions. In response to these challenges, this study introduces a novel depth estimation framework that leverages latent space features within a deep convolutional neural network to enhance the precision of monocular depth maps. The proposed model features dual encoder–decoder architecture, enabling both color-to-depth and depth-to-depth transformations. This structure allows for refined depth estimation through latent space encoding. To further improve the accuracy of depth boundaries and local features, a new loss function is introduced. This function combines latent loss with gradient loss, helping the model maintain the integrity of depth boundaries. The framework is thoroughly tested using the NYU Depth V2 dataset, where it sets a new benchmark, particularly excelling in complex indoor scenarios. The results clearly show that this approach effectively reduces depth ambiguities and blurring, making it a promising solution for applications in human–robot interaction and 3D scene reconstruction.

Distance Estimation with a Stereo Camera and Accuracy Determination

Distance measurement plays a key role in many fields of science and technology, including robotics, civil engineering, and navigation systems. This paper focuses on analyzing the precision of a measurement system using stereo camera distance measurement technology in the context of measuring two objects of different sizes. The first part of the paper presents key information about stereoscopy, followed by a discussion of the process of building a measuring station. The Mask R-CNN algorithm, which is a deep learning model that combines object detection and instance segmentation, was used to identify objects in the images. In the following section, the calibration process of the system and the distance calculation method are presented. The purpose of the study was to determine the precision of the measurement system and to identify the distance ranges where the measurements are most precise. Measurements were made in the range of 20 to 70 cm. The system demonstrated a relative error of 0.95% for larger objects and 1.46% for smaller objects at optimal distances. A detailed analysis showed that for larger objects, the system exhibited higher precision over a wider range of distances, while for smaller objects, the highest accuracy was achieved over a more limited range. These results provide valuable information on the capabilities and limitations of the measurement system used, while pointing out directions for its further optimization.

Biomimetic Active Stereo Camera System with Variable FOV

Inspired by the biological eye movements of fish such as pipefish and sandlances, this paper presents a novel dynamic calibration method specifically for active stereo vision systems to address the challenges of active cameras with varying fields of view (FOVs). By integrating static calibration based on camera rotation angles with dynamic updates of extrinsic parameters, the method leverages relative pose adjustments between the rotation axis and cameras to update extrinsic parameters continuously in real-time. It facilitates epipolar rectification as the FOV changes, and enables precise disparity computation and accurate depth information acquisition. Based on the dynamic calibration method, we develop a two-DOF bionic active camera system including two cameras driven by motors to mimic the movement of biological eyes; this compact system has a large range of visual data. Experimental results show that the calibration method is effective, and achieves high accuracy in extrinsic parameter calculations during FOV adjustments.

Towards Less Invasive Instruments For Cardiac Tissue Stabilizing

Abstract Beating-heart surgery is performed to reduce patient trauma but depending on the intervention requires tissue stabilization through vacuum cardiac tissue stabilizers. Current designs either require open chest sternotomy or display significant residual motion when used minimally invasively. Active motion compensation methods require complex control algorithms and expensive technology. We propose a novel design for a vacuum tissue stabilizer with expandable suction feet enabling a larger stabilized area and increased motion reduction. Its performance is evaluated on a heart phantom using a silicon membrane mimicking cardiac tissue properties. The motion of the membrane is captured using a stereo camera and marker points on the membrane surface. The results are then compared to a state-of-the-art tissue stabilizer. The extended stabilizer is found to stabilize a 110% larger area while achieving similar motion amplitude reduction. Thus, extendable stabilizers seem to be a promising solution for improved cardiac tissue stabilization in less invasive beating-heart surgery.

Investigating 3D Object Detection Using Stereo Camera and LiDAR Fusion with Bird's-Eye View Representation

Investigating 3D Object Detection Using Stereo Camera and LiDAR Fusion with Bird's-Eye View Representation

Humanoid Head MIKE

The humanoid robot MIKE is a PhD project developed in the Graduate Studies Section of the School of Mechanical and Electrical Engineering, Zacatenco campus of the National Polytechnic Institute (Instituto Politécnico Nacional). In this article, the head of the humanoid robot called MIKE is presented, which consists of the design of a 3D printed prototype, has 2 degrees of freedom for the movement of rotation and tilt of the head through 2 servomotors respectively, has incorporated a dual stereo camera IMX219-83 that gives it artificial vision, all this is controlled by Artificial Intelligence (AI) algorithms by means of a Raspberry Pi 5 minicomputer. To identify and classify objects, people and scenarios, programs developed in Python 3.11 were created and Open CV and RPiCam algorithms were also used. Section II presents the prototype design, head components, control system design, algorithm development and final assembly. Section III presents the tests with different scenarios and the results obtained. Section IV gives the conclusions obtained from the development of this work

Dynamic calibration and multi-view fusion of active vision tracking system for surgical instrument

Abstract In recent years, surgical instrument tracking has gained significant attention in the field of surgical navigation. An often-overlooked problem lies in the instrument’s irregular movement conflicting with the tracking system’s fixed viewpoint, leading to line-of-sight (LOS) occlusion, marker occlusion, and sub-optimal accuracy. The active vision tracking system (which actively adjusts the camera viewpoint) is applied to address this issue. In this paper, two algorithms of multi-view fusion and dynamic calibration are proposed in multi-camera active vision to allow each camera to adjust its viewpoint independently. These algorithms are integrated into the parallel computing of the frontend and backend to implement the positioning of the instrument from multiple perspectives and real-time calibration of multiple cameras. Simulations and experiments are carried out to test the accuracy and robustness of the proposed tracking system. Results reveal that our system achieves a closer trajectory (0.12 mm) to the ground truth compared to a stereo camera tracking system (0.32 mm). The proposed tracking system is able to keep track of the instruments within the positioning volume at different surgical phases, ensuring consistent navigation and improving positioning stability and accuracy.

Stereooptical Methods of Sea Surface Processes Registration

This work presents the optical methods for measurements of physical parameters on sea surface also covered by ice by using stereo cameras. New method of images with sea ice processing is offered. It can detect areas of open water and sharp edges on ice, which influence on radar signal refraction. This method is based on area of interest detection on optical images, statistical parameters for each area calculation, classification and local level definition to recognize needed structures. By using stereo system perspective distortions can be corrected and physical parameters of wind waves and sea ice can be calculated. This method was probe on stereo images obtained from 90th voyage of R/V «Akademik Mstislav Keldysh» in the Laptev Sea.The results of ripple waves velocity measurements in case on open water by using development previously stereo method is presented. Comparison with Doppler shift of microwave signal is made.The methods presented in this paper are of interest in field experiments simultaneously with the use of coherent radar stations for the quantitative interpretation of radar data, as well as for remote monitoring of the sea surface and ice conditions methods development.

Super‐Large Field‐of‐View, High‐Accurate and Real‐Time 3D Scene Reconstruction Based on Metasurface‐Enabled Structured Light

AbstractThree‐dimensional (3D) reconstruction based on structured light (SL) has wide applications in fields such as artificial intelligence, surveying and mapping, and precision manufacturing. An ideal 3D reconstruction system features both wide field‐of‐views (FOV), high‐accuracy and high‐speed reconstruction and ultracompact volume. However, these criteria have been achieved only partially with conventional SL techniques like optical projector and differentiate optical element, which will substantially limit their practical applications. In this study, an ultracompact SL and its 3D reconstruction system is designed and experimentally demonstrated, which exhibits super‐large FOV angles up to 210° × 210°, only 3.8‰ reconstruction errors, real‐time processing speeds up to 30 fps and SL dimensions as small as 0.3 mm × 0.3 mm. Specifically, the SL is realized with a resonant geometric metasurface, which can produce tens of thousands of random dots and project them into the whole 4π free space. The metasurface enabled SL is collaborated with a binocular stereoscopic camera and advanced 3D reconstruction algorism to form a practical 3D reconstruction system, which can provide robust hardware and software support for high‐accuracy and real‐time 3D scene reconstruction.

Cross-Shaped Peg-in-Hole Autonomous Assembly System via BP Neural Network Based on Force/Moment and Visual Information

Currently, research on peg-in-hole (PiH) compliant assembly is predominantly limited to circular pegs and holes, with insufficient exploration of various complex-shaped PiH tasks. Furthermore, the degree of freedom for rotation about the axis of the circular peg cannot be constrained after assembly, and few studies have covered the complete process from autonomous hole-searching to insertion. Based on the above problems, a novel cross-shaped peg and hole design has been devised. The center coordinates of the cross-hole are obtained during the hole-searching process using the three-dimensional reconstruction theory of a binocular stereo vision camera. During the insertion process, 26 contact states of the cross-peg and the cross-hole were classified, and the mapping relationship between the force-moment sensor and relative errors was established based on a backpropagation (BP) neural network, thus completing the task of autonomous PiH assembly. This system avoids hand-guiding, completely realizes the autonomous assembly task from hole-searching to insertion, and can be replaced by other structures of pegs and holes for repeated assembly after obtaining the accurate relative pose between two assembly platforms, which provides a brand-new and unified solution for complex-shaped PiH assembly.

Teleoperated forwarder with VR: impact of latency on operator log handling performance

ABSTRACT This study uses a full scale cut to length forwarder prototype to explore crane log handling during teleoperation (remote-control). Visual feedback is provided from stereo cameras into a VR headset. Details of the system are provided along with measurements on key properties. The system allows for an artificial latency (the delay between a command and the observed response) to be introduced. A setup to allow repeated crane operations is used to evaluate operator performance at different latencies and also to compare teleoperation with in-machine operation. The results indicate that teleoperation of crane log handling can achieve comparable performance to traditional in-machine operation. Notably, the performance is consistent even at transmission latencies that surpass those expected in 4 G internet connections. This suggests a potential for remotely operating forwarders from both on-site and potentially off-site locations. This research did not delve into actual forwarding work or the implications of wireless connection bandwidth or alternative technologies like Wi-Fi or 5 G. Future investigations should consider these factors, along with the requirement for enhanced camera systems to ensure holistic machine visibility and consequently, safe teleoperation.

Stereo slit-scanning tomography of the anterior segment of the human eye

We demonstrate a new tomographic imaging device for 2D cross-sectional and 3D volumetric images of the anterior eye segment. The optical system combines optical sectioning with a stereoscopic camera setup to determine the true spatial location of the ocular structures. Moreover, we propose a calibration procedure to estimate the triangulation function that links the stereo camera pixel indices with the eye-referenced 3D spatial coordinates. The proposed instrument will image the entire anterior chamber and contiguous structures, including the corneal limbus, sclera and eyelids. Furthermore, the system can obtain corneal topography. Finally, we studied the repeatability of the instrument and its agreement with a commercial Scheimpflug imaging tomographer in a group of healthy subjects.

Research on Target Hybrid Recognition and Localization Methods Based on an Industrial Camera and a Depth Camera in Complex Scenes

In order to improve the target visual recognition and localization accuracy of robotic arms in complex scenes with similar targets, hybrid recognition and localization methods based on an industrial camera and depth camera are proposed. First, according to the speed and accuracy requirements of target recognition and localization, YOLOv5s is introduced as the basic algorithm model for target hybrid recognition and localization. Then, in order to improve the accuracy of target recognition and coarse localization based on an industrial camera (eye-to-hand), the AFPN feature fusion module, simple and parameter-free attention module (SimAM), and soft non-maximum suppression (Soft NMS) are introduced. In order to improve the accuracy of target recognition and fine localization based on a depth camera (eye-in-hand), the SENetV2 backbone network structure, dynamic head module, deformable attention mechanism, and chain-of-thought prompted adaptive enhancer network are introduced. After that, on the basis of constructing a dual camera platform for target hybrid recognition and localization, the hand–eye calibration, collection and production of image datasets required for model training are completed. Finally, for the docking of the oil filling port, the hybrid recognition and localization experimental tests are completed in sequence. The test results show that in target recognition and coarse localization based on the industrial camera, the recognition accuracy of the designed model reaches 99%, and the average localization errors in the horizontal and vertical directions are 2.22 mm and 3.66 mm, respectively. In target recognition and fine localization based on the depth camera, the recognition accuracy of the designed model reaches 98%, and the average errors in depth, horizontal, and vertical directions are 0.12 mm, 0.28 mm, and 0.16 mm, respectively. These not only verify the effectiveness of the target hybrid recognition and localization methods based on dual cameras, but also demonstrate that they meet the high-precision recognition and localization requirements in complex scenes.

Passive vision based three-dimensional seam tracking using a stereo vision system with a single HDR camera

Seam tracking is a crucial feature that an automatic welding system must have. The laser tracking algorithm offers high accuracy but faces challenges in identifying seams with extremely narrow gaps, as the deformation of the laser stripe becomes difficult to detect in such cases. In contrast, the passive vision-based algorithm can effectively capture narrow gap information from the welding scene, making it more suitable for recognizing narrow seams. However, most of proposed passive vision-based seam tracking is based on “teaching and playback” mode and can only realize planar tracking. In medium and thick steel plate welding, the workpieces are so large that the “teaching and playback” based control algorithm is unsuitable. Therefore, in this paper, a stereo vision system with a single camera and the corresponding calibration algorithm are developed to achieve three dimensional seam tracking. Compared to stereo system with dual camera, the proposed system is cheaper and easier to synchronize acquisition. Moreover, to get out of the “teaching and playback” control strategy, a vector control algorithm without cumulative error is proposed. Results of nine experiments show that the average error of the proposed seam tracking system is less than 1.0662mm and the maximum error is 1.8030mm.

Point cloud segmentation method based on an image mask and its application verification

Abstract Accurately perceiving three-dimensional (3D) environments or objects is crucial for the advancement of artificial intelligence interaction technologies. Currently, various types of sensors are employed to obtain point cloud data for 3D object detection or segmentation tasks. While this multi-sensor approach provides more precise 3D data than monocular or stereo cameras, it is also more expensive. The advent of RGB-D cameras, which provide both RGB images and depth information, addresses this issue. In this study, we propose a point cloud segmentation method based on image masks. By using an RGB-D camera to capture color and depth images, we generate image masks through object recognition and segmentation. Given the mapping relationship between RGB image pixels and point clouds, these image masks can be further used to extract the point cloud data of the target objects. The experimental results revealed that the average accuracy of target segmentation was 84.78%, which was close to that of PointNet++. Compared with three traditional segmentation algorithms, the accuracy was improved by nearly 23.97%. The running time of our algorithm is reduced by 95.76% compared to the PointNet++ algorithm, which has the longest running time; and by 15.65% compared to the LCCP algorithm, which has the shortest running time among traditional methods. Compared with PointNet++, the segmentation accuracy was improved. This method addressed the issues of low robustness and excessive reliance on manual feature extraction in traditional point cloud segmentation methods, providing valuable support and reference for the accurate segmentation of 3D point clouds.

New metrics for measuring 2D uniformity in stirring system based on reconstruction of the particle trajectory

In industrial mixing systems, the evaluation method is a crucial indicator for measuring mixing uniformity. This study proposes an assessment method that combines a method for positioning using dual cameras and a point pattern density fluctuation (PD) method based on disordered hyperuniformity. This study employs a method for positioning using dual cameras to investigate the motion trajectories of tracer balls and achieve a 3D reconstruction process. The reconstructed 3D systems are cross-sectioned through on the axis of the stirring paddle to form a 2D planar diagram of the tracer ball trajectories. The PD method is used to evaluate the mixing effectiveness of the 2D trajectory points, establishing evaluation metrics for the 2D mixing system. To ensure the accuracy of the PD method, a feasibility analysis was conducted using the conductometry. A relationship model between λ and mixing time was established, with a Pearson correlation coefficient (r) of -0.9867, indicating a strong negative linear correlation. The relationship between the planes and whole 3D space was analyzed in terms of the whole and the discrete perspective, as well as the existing methods were compared with other approaches. This study not only deepens the theoretical understanding of evaluating mixing uniformity but also provides a crucial foundation and theoretical basis for process optimization of stirring systems and design of stirred reactors. It offers valuable references and insights for engineering design and technological improvements in related fields.

Fringe photometric stereo

In fringe projection profilometry (FPP), a high spatial frequency of fringe typically indicates powerful error suppression capability; however, it complicates phase unwrapping, necessitating a greater number of patterns and thus compromising the real-time performance of scanning. In this letter, we report a fringe photometric stereo (FPS) to completely bypass phase unwrapping for recovering the continuous 3D surface. We reveal the linear mapping relationship between the phase gradient and depth gradient, thereby facilitating the computation of depth gradients from phase gradients. Thus, we can employ depth gradients to reconstruct high-quality 3D profiles. Experimental results demonstrate that our FPS surpasses traditional phase-shifting profilometry in mitigating Gaussian noise. Our approach enables high-quality and unambiguous 3D reconstruction using single-frequency fringe patterns, relying solely on a camera and a projector without the need for dual, multiple, or light field cameras, thereby paving a path to high-speed and precision 3D imaging.

Visual-Inertial Fusion-Based Five-Degree-of-Freedom Motion Measurement System for Vessel-Mounted Cranes

Vessel-mounted cranes operate in complex marine environments, where precise measurement of cargo positions and attitudes is a key technological challenge to ensure operational stability and safety. This study introduces an integrated measurement system that combines vision and inertial sensing technologies, utilizing a stereo camera and two inertial measurement units (IMUs) to capture cargo motion in five degrees of freedom (DOF). By merging data from the stereo camera and IMUs, the system accurately determines the cargo’s position and attitude relative to the camera. The specific methodology is introduced as follows: First, the YOLO model is adopted to identify targets in the image and generate bounding boxes. Then, using the principle of binocular disparity, the depth within the bounding box is calculated to determine the target’s three-dimensional position in the camera coordinate system. Simultaneously, the IMU measures the attitude of the cargo, and a Kalman filter is applied to fuse the data from the two sensors. Experimental results indicate that the system’s measurement errors in the x, y, and z directions are less than 2.58%, 3.35%, and 3.37%, respectively, while errors in the roll and pitch directions are 3.87% and 5.02%. These results demonstrate that the designed measurement system effectively provides the necessary motion information in 5-DOF for vessel-mounted crane control, offering new approaches for pose detection of marine cranes and cargoes.

Pixel-level Terrain Processing of the Martian Surface from Moderate Resolution Images of Tianwen-1

The moderate-resolution camera (MoRIC) of the Tianwen-1 orbiter, China’s first Mars exploration mission, is a frame-imaging scheme color camera. However, generating a precise digital elevation model (DEM) from these images faces challenges due to the presence of featureless and repetitive regions on the Martian surface. Traditional photogrammetric methods often struggle to achieve pixel-level resolution in these areas. To address this, we employ a shape from shading (SFS) algorithm that leverages shading patterns to reconstruct fine-scale terrain details, optimizing photogrammetric initial DEM. This paper presents a novel global bisection search algorithm, based on the Pearson correlation coefficient to determine the smoothing parameter, a crucial element in the SFS process. Comparisons with higher-resolution data from the Mars Express high-resolution stereo camera and Mars Reconnaissance Orbiter CTX images validate our approach, highlighting the potential of SFS to significantly enhance the scientific value of Tianwen-1 MoRIC data by generating high-resolution, three-dimensional maps of Mars.