Epipolar Lines Research Articles

Adaptive Feature Enhanced Multi-View Stereo With Epipolar Line Information Aggregation

Adaptive Feature Enhanced Multi-View Stereo With Epipolar Line Information Aggregation

Multi-line laser 3D reconstruction method based on spatial quadric surface and geometric estimation

The traditional multi-line laser 3D reconstruction is based on binocular epipolar constraints and the equation of the laser optical plane. Firstly, matching points are identified based on the epipolar constraints. Then, the correct matching points are selected based on the optical plane equation of the multi-line laser. Finally, 3D reconstruction is performed using the selected matching points. In actual production processes involving multi-line lasers, noticeable distortion occurs due to the projection of Diffractive Optical Elements (DOE) at a large angle. This results in curved laser light planes and curved reflections on the measured objects. Under such circumstances, efficiently screening matching points using the traditional method based on the laser plane equation becomes challenging. Additionally, inherent noise in multi-line laser systems introduces errors in extracted laser center coordinates, making it impossible to directly obtain high-precision 3D data. To address these issues, this paper proposes a method that utilizes spatial quadric surfaces and geometric estimation for completing multi-line laser 3D reconstructions. By analyzing the distortion principle of DOE and the positional relationship after optical diffraction, the multi-line laser manifests as a quadric surface on the optical output plane. Consequently, by calibrating the equations of the quadric surface and applying binocular polar constraints, suitable matching points for the multi-line laser can be chosen. Once an accurate matching point is identified, a minimum geometric distance estimation can be established based on the distance constraint between the point and its corresponding epipolar line. This distance represents the separation between the left and right camera’s laser center points and their respective epipolar lines. Utilizing this estimate of the minimum geometric distance, a refined calculation can be performed to better satisfy epipolar constraints and obtain new matching points. Ultimately, utilizing these new matching points enables the completion of 3D reconstruction for multi-line lasers. In comparison with methods relying on spatial plane and epipolar constraints, our algorithm exhibits a superior degree of matchability and accuracy.According to multiple groups of test data, the average error before optimization is 0.3126 mm, and can be increased to 0.0336 mm after optimization.

High-accuracy fringe projection profilometry without phase unwrapping based on multi-view geometry constraints

The number of fringes and phase unwrapping in fringe projection profilometry result in two key factors. The first is to avoid the problems of excessive fringe patterns, and the second is phase ambiguity. This paper presents a three-dimensional (3D) measurement method without phase unwrapping. This method benefits from the geometric constraints and does not require additional images. Meanwhile, epipolar rectification is performed to calibrate the rotation matrix relationship between the new plane of the dual camera and the plane of the projector. Subsequently, using depth constraints, the point pairs with incorrect 3D positions are effectively eliminated, and the initial parallax map is obtained by establishing epipolar lines of the left and right matching points in the projector domain, obtaining the intersection points, and setting up the threshold for filtering. Finally, a function combining the modulation intensity and phase is proposed to refine the parallax map such that the 3D result is insensitive to phase error. The standard step block and standard ball were used to verify the validity of the proposed method, and the experimental results showed that the root mean square error of the method was 0.052 mm.

Noise-Robust 3D Pose Estimation Using Appearance Similarity Based on the Distributed Multiple Views.

In this paper, we present a noise-robust approach for the 3D pose estimation of multiple people using appearance similarity. The common methods identify the cross-view correspondences between the detected keypoints and determine their association with a specific person by measuring the distances between the epipolar lines and the joint locations of the 2D keypoints across all the views. Although existing methods achieve remarkable accuracy, they are still sensitive to camera calibration, making them unsuitable for noisy environments where any of the cameras slightly change angle or position. To address these limitations and fix camera calibration error in real-time, we propose a framework for 3D pose estimation which uses appearance similarity. In the proposed framework, we detect the 2D keypoints and extract the appearance feature and transfer it to the central server. The central server uses geometrical affinity and appearance similarity to match the detected 2D human poses to each person. Then, it compares these two groups to identify calibration errors. If a camera with the wrong calibration is identified, the central server fixes the calibration error, ensuring accuracy in the 3D reconstruction of skeletons. In the experimental environment, we verified that the proposed algorithm is robust against false geometrical errors. It achieves around 11.5% and 8% improvement in the accuracy of 3D pose estimation on the Campus and Shelf datasets, respectively.

Fast phase matching based on order partition with sub-pixel accuracy

Structured light is a method of 3D imaging that involves projecting a sequence of striped or structured patterns onto a target scene and then, based on the warping of stripes across a target surface, a camera can reconstruct the target. For practical reasons relating to changing lens parameters for the projector, binocular structured light improves over monocular structured by performing triangulation between two cameras, using the striped patterns to solve the correspondence matching problem; however, the speed at which these correspondences are found is still an active area of research. And in this paper, we propose a fast-matching algorithm that enhances phase labeling and introduces the concept of triangle plane interpolation based on linear interpolation. The experimental results demonstrate that our method achieves a noticeable speed improvement in computational efficiency and a stable improvement in accuracy as compared to the phase matching along epipolar line pointwisely based on look-up tables.

Dynamic Validation of Calibration Accuracy and Structural Robustness of a Multi-Sensor Mobile Robot.

For mobile robots, the high-precision integrated calibration and structural robustness of multi-sensor systems are important prerequisites for ensuring healthy operations in the later stage. Currently, there is no well-established validation method for the calibration accuracy and structural robustness of multi-sensor systems, especially for dynamic traveling situations. This paper presents a novel validation method for the calibration accuracy and structural robustness of a multi-sensor mobile robot. The method employs a ground-object-air cooperation mechanism, termed the "ground surface simulation field (GSSF)-mobile robot -photoelectric transmitter station (PTS)". Firstly, a static high-precision GSSF is established with the true north datum as a unified reference. Secondly, a rotatable synchronous tracking system (PTS) is assembled to conduct real-time pose measurements for a mobile vehicle. The relationship between each sensor and the vehicle body is utilized to measure the dynamic pose of each sensor. Finally, the calibration accuracy and structural robustness of the sensors are dynamically evaluated. In this context, epipolar line alignment is employed to assess the accuracy of the evaluation of relative orientation calibration of binocular cameras. Point cloud projection and superposition are utilized to realize the evaluation of absolute calibration accuracy and structural robustness of individual sensors, including the navigation camera (Navcam), hazard avoidance camera (Hazcam), multispectral camera, time-of-flight depth camera (TOF), and light detection and ranging (LiDAR), with respect to the vehicle body. The experimental results demonstrate that the proposed method offers a reliable means of dynamic validation for the testing phase of a mobile robot.

A dense matching method for remote sensing images fused with CPS denoising

Dense matching of remote sensing images is crucial for 3D reconstruction. This study proposes an enhanced dense matching method employing the CPS image denoising algorithm, aiming to boost the SGM algorithm's accuracy and efficiency in remote sensing image matching. The stereo image pair's quality is evaluated using the PSNR index, and a decision-making criterion based on the CPS algorithm is incorporated to determine the need for denoising. Preprocessing steps, including image cropping and pixel coordinate transformation, significantly reduce computational requirements. An epipolar line model, minimizing the disparity between two pixels, is used for calculations. This model is employed to construct an epipolar image, enhancing the accuracy and efficiency of the process. The study conducted experimental validation and analysis of the mismatch rate, running time, and denoising effect of the algorithm using the Middlebury 2021 stereo datasets. Additionally, the matching results of the World-View3 satellite stereo image pairs were visualized and analyzed. The experimental results indicate that the proposed algorithm reduces the average mismatch rate by 13.1% and increases the running speed by about 3 to 4 times compared to the SGBM algorithm. Specifically, the denoising effect reduces the mismatch rate of the reconstructed image by an average of 8.97%. The results indicate that the CPS method effectively addresses dense matching challenges in the presence of image blur and noise, thereby improving the operational efficiency and accuracy of the dense matching algorithm.

A digital speckle stereo matching algorithm based on epipolar line correction

A digital speckle stereo matching algorithm based on epipolar line correction

U-ETMVSNet: Uncertainty-Epipolar Transformer Multi-View Stereo Network for Object Stereo Reconstruction

The Multi-View Stereo model (MVS), which utilizes 2D images from multiple perspectives for 3D reconstruction, is a crucial technique in the field of 3D vision. To address the poor correlation between 2D features and 3D space in existing MVS models, as well as the high sampling rate required for static sampling, we proposeU-ETMVSNet in this paper. Initially, we employ an integrated epipolar transformer module (ET) to establish 3D spatial correlations along epipolar lines, thereby enhancing the reliability of aggregated cost volumes. Subsequently, we devise a sampling module based on probability volume uncertainty to dynamically adjust the depth sampling range for the next stage. Finally, we utilize a multi-stage joint learning method based on multi-depth value classification to evaluate and optimize the model. Experimental results demonstrate that on the DTU dataset, our method achieves a relative performance improvement of 27.01% and 11.27% in terms of completeness error and overall error, respectively, compared to CasMVSNet, even at lower depth sampling rates. Moreover, our method exhibits excellent performance with a score of 58.60 on the Tanks &Temples dataset, highlighting its robustness and generalization capability.

RC-SLAM: Road Constrained Stereo Visual SLAM System Based on Graph Optimization.

Intelligent vehicles are constrained by road, resulting in a disparity between the assumed six degrees of freedom (DoF) motion within the Visual Simultaneous Localization and Mapping (SLAM) system and the approximate planar motion of vehicles in local areas, inevitably causing additional pose estimation errors. To address this problem, a stereo Visual SLAM system with road constraints based on graph optimization is proposed, called RC-SLAM. Addressing the challenge of representing roads parametrically, a novel method is proposed to approximate local roads as discrete planes and extract parameters of local road planes (LRPs) using homography. Unlike conventional methods, constraints between the vehicle and LRPs are established, effectively mitigating errors arising from assumed six DoF motion in the system. Furthermore, to avoid the impact of depth uncertainty in road features, epipolar constraints are employed to estimate rotation by minimizing the distance between road feature points and epipolar lines, robust rotation estimation is achieved despite depth uncertainties. Notably, a distinctive nonlinear optimization model based on graph optimization is presented, jointly optimizing the poses of vehicle trajectories, LPRs, and map points. The experiments on two datasets demonstrate that the proposed system achieved more accurate estimations of vehicle trajectories by introducing constraints between the vehicle and LRPs. The experiments on a real-world dataset further validate the effectiveness of the proposed system.

Light Field Neural Rendering.

Classical light field rendering for novel view synthesis can accurately reproduce view-dependent effects such as reflection, refraction, and translucency, but requires a dense view sampling of the scene. Methods based on geometric reconstruction need only sparse views, but cannot accurately model non-Lambertian effects. We introduce a model that combines the strengths and mitigates the limitations of these two directions. By operating on a four-dimensional representation of the light field, our model learns to represent view-dependent effects accurately. By enforcing geometric constraints during training and inference, the scene geometry is implicitly learned from a sparse set of views. Concretely, we introduce a two-stage transformer-based model that first aggregates features along epipolar lines, then aggregates features along reference views to produce the color of a target ray. Additionally, we propose modifications that allow the model to generalize to scenes without any fine-tuning. Our model outperforms the state-of-the-art on multiple forward-facing and 360 ° datasets, with larger margins on scenes with severe view-dependent variations. Code and results can be found at https://light-field-neural-rendering.github.io/.

Calibration of Fisheye Camera with Co-linear Constraint of the Main Optical Axis and Epipolar Line Orthogonality-Transverse Axis

Calibration of Fisheye Camera with Co-linear Constraint of the Main Optical Axis and Epipolar Line Orthogonality-Transverse Axis

UAV image matching of mountainous terrain using the LoFTR deep learning model

In natural terrain scene UAV image matching, traditional feature point-based methods often have problems such as an unstable number of extracted feature points, difficulty in detecting feature points in weak texture areas, uneven distribution, and low robustness. Deep learning-based image matching methods can produce larger and more reasonably distributed matching pairs, so this research paper tries to perform UAV image matching based on a deep learning LoFTR algorithm for natural terrain scenes. The critical technical process was: first, the LoFTR algorithm was used to generate dense feature matching, and then the epipolar line constraints were used to purify the interior points, specifically, this study used the MAGSAC++ method to estimate the fundamental matrix, eliminate the wrong matching pairs, and finally get reliable matching results. In this research paper, six sets of visible images taken by different UAVs equipped with different sensors in the field were selected as experimental data to test the method and were compared and analyzed with the traditional classical SIFT, ASIFT, and AKAZE algorithms and the KeyNet-AdaLAM deep learning method. The experimental results show that the method in this study obtains a dense number of robust matching pairs with uniform spatial distribution in the UAV image matching of natural scenes mainly in mountainous areas, and the comprehensive performance is higher and more advantageous than the comparison methods.

Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity

Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume and may fail when the range is too large or unreliable. To address this problem, we propose a disparity-based MVS method based on the epipolar disparity flow (E-flow), called DispMVS, which infers the depth information from the pixel movement between two views. The core of DispMVS is to construct a 2D cost volume on the image plane along the epipolar line between each pair (between the reference image and several source images) for pixel matching and fuse uncountable depths triangulated from each pair by multi-view geometry to ensure multi-view consistency. To be robust, DispMVS starts from a randomly initialized depth map and iteratively refines the depth map with the help of the coarse-to-fine strategy. Experiments on DTUMVS and Tanks\&Temple datasets show that DispMVS is not sensitive to the depth range and achieves state-of-the-art results with lower GPU memory.

High-precision pose measurement method based on binocular vision in dark lighting environments

Measuring the pose of non-cooperative targets in space is a critical supporting technology for cleaning up space debris and recovering items. However, most existing methods are simulation experiments conducted in good lighting environments and tend to show poor performance in dark lighting environments. A target pose measurement method based on binocular vision is proposed, which is suitable for dark lighting environments. First, the traditional features from accelerated segment test algorithm are improved to reduce the influence of illumination on the performance of feature point extraction under various postures. The point feature and line feature are combined to extract image features more easily in a dark lighting environment while retaining the high accuracy of the pose measurement algorithm based on point features. Second, the normalized cross-correlation coefficient matching method is combined with the epipolar constraint to narrow the search range of the matching points from the two-dimensional plane to the epipolar line, which substantially improves the matching efficiency and accuracy of the matching algorithm. Finally, post-processing through feature matching is performed to reduce the probability of mismatches. Simulation and physical experiment results show that our method can stably extract features and obtain high-precision target pose information in well-illuminated as well as dark lighting environments, making it suitable for high-precision target pose measurement under insufficient illumination.

SOFT2: Stereo Visual Odometry for Road Vehicles Based on a Point-to-Epipolar-Line Metric

Accurate localization constitutes a fundamental building block of any autonomous system. In this article, we focus on stereo cameras and present a novel approach, dubbed SOFT2, that is currently the highest-ranking algorithm on the KITTI scoreboard. SOFT2 relies on the constraints imposed by the epipolar geometry and kinematics, i.e., it is developed for configurations that cannot exhibit pure rotation. We minimize point-to-epipolar-line distances, which makes the approach resilient to object depth uncertainty, and as the first step, we estimate motion up to scale using just a single camera. Then, we propose to jointly estimate the absolute scale and the extrinsic rotation of the second camera in order to alleviate the effects of varying stereo rig extrinsics. Finally, we smooth the motion estimates in a temporal window of frames by using the proposed epipolar line bundle adjustment procedure. We also introduce a multiple hypothesis feature-matching approach for self-similar planar surfaces that account for appearance change due to perspective. We evaluate SOFT2 and compare it to ORB-SLAM2, OV2SLAM, and VINS-FUSION on the KITTI-360 dataset, KITTI train sequences, Málaga Urban dataset, Oxford Robotics Car dataset, and Multivehicle Stereo Event Camera dataset.

RayMVSNet++: Learning Ray-Based 1D Implicit Fields for Accurate Multi-View Stereo.

Learning-based multi-view stereo (MVS) has by far centered around 3D convolution on cost volumes. Due to the high computation and memory consumption of 3D CNN, the resolution of output depth is often considerably limited. Different from most existing works dedicated to adaptive refinement of cost volumes, we opt to directly optimize the depth value along each camera ray, mimicking the range (depth) finding of a laser scanner. This reduces the MVS problem to ray-based depth optimization which is much more light-weight than full cost volume optimization. In particular, we propose RayMVSNet which learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. This sequential modeling, conducted based on transformer features, essentially learns the epipolar line search in traditional multi-view stereo. We devise a multi-task learning for better optimization convergence and depth accuracy. We found the monotonicity property of the SDFs along each ray greatly benefits the depth estimation. Our method ranks top on both the DTU and the Tanks & Temples datasets over all previous learning-based methods, achieving an overall reconstruction score of 0.33 mm on DTU and an F-score of 59.48% on Tanks & Temples. It is able to produce high-quality depth estimation and point cloud reconstruction in challenging scenarios such as objects/scenes with non-textured surface, severe occlusion, and highly varying depth range. Further, we propose RayMVSNet++ to enhance contextual feature aggregation for each ray through designing an attentional gating unit to select semantically relevant neighboring rays within the local frustum around that ray. This improves the performance on datasets with more challenging examples (e.g., low-quality images caused by poor lighting conditions or motion blur). RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset. In particular, it attains an AbsRel of 0.058m and produces accurate results on the two subsets of textureless regions and large depth variation.

Parallel optimization of tridimensional deformation measurement based on correlation function constraints of a multi-camera network.

This study primarily investigates the low accuracy and redundant time-consuming problem of speckle registration in the full-field deformation measurement of slender and biggish specimens. To solve these problems, a parallel optimization of the tridimensional deformation measurement method is proposed based on what we believe is a novel correlation function constraints of a multi-camera network. First, a neotype correlation function is built based on the joint constraint relationship among the multiple cameras, which is capable of accurately restricting the search for homologous points in image pairs to the epipolar line, instead of the entire image, while significantly narrowing the search space and accelerating the search. The multiple cameras are bundled as a whole, thus reducing the dimension of the Jacobian matrix and the normalized matrix to a certain extent. Subsequently, more speckle images can be calculated in one iteration. Furthermore, the decomposition of the derived correlation function and the scheme of the parallel algorithm are decomposed via the kernel function based on the GPU parallel mechanism of the compute unified device architecture source program, thus increasing the subpixel search speed of speckle matching and ensuring the calculation performance of the stereo deformation measurement method to reach a higher level. Lastly, the experimental results revealed that the proposed strategy could allow the calculation speed-up ratio of speckle sequence and stereo registration to reach 20.390 times and 17.873 times, respectively, while ensuring the out-of-plane displacement average measuring accuracy to be higher than 0.179mm within the spatial range of [2m, 2m, 3m]. As a result, the proposed approach has crucial applications in rapid and stable tridimensional deformation measurement.

Adaptive Modulation and Rectangular Convolutional Network for Stereo Image Super-Resolution

• We propose a novel network for better stereo image super-resolution. • We propose an adaptive modulation alignment mechanism to modulate the aligned features. • We add rectangular convolutional layers to expand the receptive field along the disparity epipolar line . • Experimental results show that our method outperforms other state-of-the-art methods on the three stereo benchmarks. Deep learning based stereo image super-resolution algorithm can make use of the additional information in the stereo image pairs to improve the quality of the reconstructed images. However, extracting the similarity information of one image to another precisely is challenging when partially occluded regions exist in stereo images. In this paper, we propose an adaptive modulation alignment mechanism to modulate the aligned features calculated from the parallax attention mechanism and effectively deal with the inaccuracy caused by occlusion. Furthermore, because disparities of stereo image exist only along the epipolar line, we use rectangular convolution kernel in some convolution layers , to extend the receptive field horizontally. Finally, experimental results demonstrate that our method achieves state-of-the-art performance on the Middlebury, KITTI 2012 and KITTI 2015 stereo benchmarks.

Fringe projection profilometry method with high efficiency, precision, and convenience: theoretical analysis and development.

Fringe projector profilometry (FPP) is an important three-dimensional (3D) measurement technique, especially when high precision and speed are required. Thus, theoretical interrogation is critical to provide deep understanding and possible improvement of FPP. By dividing an FPP measurement process into four steps (system calibration, phase measurement, pixel correspondence, and 3D reconstruction), we give theoretical analysis on the entire process except for the extensively studied calibration step. Our study indeed reveals a series of important system properties, to the best of our knowledge, for the first time: (i) in phase measurement, the optimal and worst fringe angles are proven perpendicular and parallel to epipolar line, respectively, and can be considered as system parameters and can be directly made available during traditional calibration, highlighting the significance of the epipolar line; (ii) in correspondence, when two sets of fringes with different fringe orientations are projected, the highest correspondence precision can be achieved with arbitrary orientations as long as these two orientations are perpendicular to each other; (iii) in reconstruction, a higher reconstruction precision is given by the 4-equation methods, while we notice that the 3-equation methods are almost dominatingly used in literature. Based on these theoretical results, we propose a novel FPP measurement method which (i) only projects one set of fringes with optimal fringe angle to explicitly work together with the epipolar line for precise pixel correspondence; (ii) for the first time, the optimal fringe angle is determined directly from the calibration parameters, instead of being measured; (iii) uses 4 equations for precise 3D reconstruction but we can remove one equation which is equivalent to an epipolar line, making it the first algorithm that can use 3-equation solution to achieve 4-equation precision. Our method is efficient (only one set of fringe patterns is required in projection and the speed is doubled in reconstruction), precise (in both pixel correspondence and 3D reconstruction), and convenient (the computable optimal fringe angle and a closed-form 3-equation solution). We also believe that our work is insightful in revealing fundamental FPP properties, provides a more reasonable measurement for practice, and thus is beneficial to further FPP studies.