Epipolar Research Articles

Light-field angular super-resolution reconstruction via fusing 3D epipolar plane images

Light-field angular super-resolution reconstruction via fusing 3D epipolar plane images

An Epipolar HS-NCC Flow Algorithm for DSM Generation Using GaoFen-3 Stereo SAR Images

Radargrammetry is a widely used methodology to generate the large-scale Digital Surface Model (DSM). Stereo matching is the most challenging step in radargrammetry due to the significant geometric differences and the inherent speckle noise. The speckle noise results in significant grayscale differences of the same feature points, which makes the traditional Horn–Schunck (HS) flow or multi-window zero-mean normalized cross-correlation (ZNCC) methods degrade. Therefore, this paper proposes an algorithm named Epipolar HS-NCC Flow (EHNF) for dense stereo matching, which is an improved HS flow method with normalized cross-correction constraint based on epipolar stereo images. First, the epipolar geometry is applied to resample the image to realize the coarse stereo matching. Subsequently, the EHNF method forms a global energy function to achieve fine stereo matching. The EHNF method constructs a local normalized cross-correlation constraint term to compensate for the grayscale invariance constraint, especially for the SAR stereo images. Additionally, two assessment methods are proposed to calculate the optimal cross-correlation parameter and smoothness parameter according to the refined matched point pairs. Two GaoFen-3 (GF-3) image pairs from ascending and descending orbits and the open Light Detection and Ranging (LiDAR) data are utilized to fully evaluate the proposed method. The results demonstrate that the EHNF algorithm improves the DSM elevation accuracy by 9.6% and 27.0% compared with the HS flow and multi-window ZNCC methods, respectively.

A high-accuracy circular hole measurement method based on multi-camera system

Industrial parts often contain a large number of circular holes that are often used as assembly datums. Measuring circular holes with high accuracy is very challenging due to the low signal-to-noise ratio of the edges on the image. This problem is further complicated when chamfers and nonuniform reflections are present. In this paper, a high-accuracy method for measuring circular holes is proposed to realize on-line inspection under a multi-camera system. In the image processing stage, an ellipse detector is designed based on a high-confidence edge detection algorithm and state-of-the-art ellipse fitting technology to significantly eliminate the ellipse fitting bias caused by blurred edges. In the ellipse matching and validation stage, the ambiguous objects are reconstructed first using ambiguous ellipse pairs matched by the epipolar geometry. Then, the similarity between the reprojected ellipse and the detected ellipse is assessed by calculating the Euclidean distance to significantly eliminate mismatched candidates and select the best matching ones. Subsequently, the discrete 3-D points are calculated by spatial intersection after obtaining the homologous points on the corresponding ellipses. Furthermore, a robust spatial circle fitting method is developed to adapt to discrete 3-D points that are not strictly located on a plane. Finally, extensive experiments on both real-world datasets and simulation datasets demonstrate that the proposed algorithm is beneficial for improving the measurement accuracy compared with the state-of-the-art algorithms and demonstrate great potential for industrial applications.

TMSO-Net: Texture adaptive multi-scale observation for light field image depth estimation

Light field can record the four-dimensional information of light rays, i.e. the position and direction information in which depth information is implied. To improve the depth estimation accuracy, we propose a depth estimation algorithm based on convolutional neural network (CNN). First, a single image super resolution algorithm is adopted to spatially super resolve the sub-aperture images (SAIs). Second, to adapt the texture complexity, the SAIs are partitioned into two regions, i.e., simple texture region and complex texture region, based on the texture analysis of the central SAI. Third, the epipolar plane images (EPIs) in horizontal, vertical, 45 degree diagonal, and 135 degree diagonal directions for both complex and simple texture regions are extracted, and the corresponding EPIs for the simple and complex texture regions are fed into the specified network branches. Finally, a fusion module is designed to generate the depth map. Experimental results show that the quality of the estimated depth maps by the proposed method is better than the state-of-the-art methods in terms of both objective quality and subjective quality. Moreover, the proposed method is more robust to noise.

Unsupervised Monocular Visual Odometry for Fast-Moving Scenes Based on Optical Flow Network with Feature Point Matching Constraint.

Robust and accurate visual feature tracking is essential for good pose estimation in visual odometry. However, in fast-moving scenes, feature point extraction and matching are unstable because of blurred images and large image disparity. In this paper, we propose an unsupervised monocular visual odometry framework based on a fusion of features extracted from two sources, that is, the optical flow network and the traditional point feature extractor. In the training process, point features are generated for scene images and the outliers of matched point pairs are filtered by FlannMatch. Meanwhile, the optical flow network constrained by the principle of forward-backward flow consistency is used to select another group of corresponding point pairs. The Euclidean distance between the matching points found by FlannMatch and the corresponding point pairs by the flow network is added to the loss function of the flow network. Compared with SURF, the trained flow network shows more robust performance in complicated fast-motion scenarios. Furthermore, we propose the AvgFlow estimation module, which selects one group of the matched point pairs generated by the two methods according to the scene motion. The camera pose is then recovered by Perspective-n-Point (PnP) or the epipolar geometry. Experiments conducted on the KITTI Odometry dataset verify the effectiveness of the trajectory estimation of our approach, especially in fast-moving scenarios.

Circular fringe projection profilometry and 3D sensitivity analysis based on extended epipolar geometry

For fringe projection profilometry, the phase-shifting direction needs to fit the position layout of a camera and a projector, or the quality of 3D reconstruction will be poor. Two-direction fringe projection profilometry, i.e. scanning along both horizontal and vertical directions, achieves robust performance, while the number of patterns is doubled comparing with one-direction scanning. In this paper, based on the extended epipolar geometry, we propose circular fringe projection profilometry to overcome the direction issue without doubling the number of patterns. First, by introducing the farthest and nearest points to the epipole in projector space, we can robustly control the spatial frequency of circular fringes in patterns. Second, by analytically relating the circularly-shaped phase to 3D world coordinate, we naturally reduce the number of the equations from 6 down to 3 for the proposed algorithm, and compute 3D point clouds in real time. Finally, we propose 3D sensitivity to conveniently analyze the error of reconstruction point clouds. The experimental results show that, compared with two-direction fringe projection profilometry which is robust in accuracy, the proposed is of a little better quality while the number of the patterns is half.

Immediate Vehicle Movement Estimation and 3D Reconstruction for Mono Cameras by Utilizing Epipolar Geometry and Direction Prior

Motion estimation of surrounding objects is indispensable to any mobile machinery. The paper proposes a method to solve the estimation and reconstruction problem of dynamic objects with a mono camera. Using the relative camera motion and detected rigidly moving objects on the image, we estimate their movement up to a scale factor. Utilization priors about their moving direction are used to estimate the transformation, which maps the 3D object from the previous frame to the actual one. Our two-frame method works twice the speed or more as other methods using three frames or more for the estimation, and we do this without any constraints. We evaluate our method on various traffic scenarios of different autonomous driving datasets.

Real-Time Stereo Visual Odometry Based on an Improved KLT Method

Real-time stereo visual odometry (SVO) localization is a challenging problem, especially for a mobile platform without parallel computing capability. A possible solution is to reduce the computational complexity of SVO using a Kanade–Lucas–Tomasi (KLT) feature tracker. However, the standard KLT is susceptible to scale distortion and affine transformation. Therefore, this work presents a novel SVO algorithm yielding robust and real-time localization based on an improved KLT method. First, in order to improve real-time performance, feature inheritance is applied to avoid time-consuming feature detection and matching processes as much as possible. Furthermore, a joint adaptive function with respect to the average disparity, translation velocity, and yaw angle is proposed to determine a suitable window size for the adaptive KLT tracker. Then, combining the standard KLT method with an epipolar constraint, a simplified KLT matcher is introduced to substitute feature-based stereo matching. Additionally, an effective veer chain matching scheme is employed to reduce the drift error. Comparative experiments on the KITTI odometry benchmark show that the proposed method achieves significant improvement in terms of time performance than the state-of-the-art single-thread approaches and strikes a better trade-off between efficiency and accuracy than the parallel SVO or multi-threaded SLAM.

Study on Optimization Method of Visual Odometry Based on Feature Matching

The mismatching of image features affects the calculation of the fundamental matrix and then leads to poor estimation accuracy of SLAM visual odometry. Aiming at the above problems, a visual odometry optimization method based on feature matching is proposed. Firstly, the initial matching set is roughly filtered by the minimum distance threshold method, and then, the relative transformation relationship between images is calculated by the RANSAC algorithm. If it conforms to the transformation relationship, it is an interior point. The iteration result with most interior points is the correct matching result. Then, the homography transformation between images is calculated, and the fundamental matrix is calculated by it. The interior points are determined by epipolar geometric constraints, and the fundamental matrix with the most interior points is obtained. Finally, the effects of the visual odometry optimization algorithm are verified by the TUM data set from two aspects: feature matching and fundamental matrix calculation. The experimental results show that an improved feature matching algorithm can effectively remove mismatched feature points while improving the operation efficiency. At the same time, the accuracy of feature point matching is increased by 15.8%. The fundamental matrix estimation algorithm not only improves the calculation accuracy of the fundamental matrix but also increases the interior point rate by 11.9%. A theoretical basis for improving the accuracy estimation of visual odometry will be provided.

Epipolar Geometry‐Based Visual Servoing of Soft Endoscopic Manipulator for Transoral Laser Ablation

Soft manipulators integrated with optical laser fibers offer new opportunities for endoscopic noncontact laser surgery. To achieve precise laser projection in a confined workspace and avoid damage, a controller with high accuracy and stability is required. An effective way is to close the control loop. Therefore, a visual servo controller that allows automatic laser spot steering using soft manipulators is proposed. An epipolar geometry model is established to acquire the inverse transition mapping from image to actuation. With this mapping, a feedback controller is derived without prior information of tissue surface geometry. Experimental validation demonstrates accurate path following using a magnetic resonance‐safe manipulator, with root‐mean‐square (RMS) tracking errors <4 pixels (140 μm) in the camera view. This is maintained even throughout 70 repeated cycles, with a maximum tracking error <11.32 pixels (≈396.2 μm). The controller enables laser spot following of the mouse cursor with an RMS tracking error of 3.97 pixels (≈180.5 μm). Ex vivo tissue ablation tests are conducted to validate the laser ablation pattern, with an RMS tracking error of about 5.75 pixels (≈201.3 μm). A laser ablation test is conducted under magnetic resonance imaging (MRI) to validate the feasibility of MRI‐guided laser surgery.

Stereo camera pair calibration for traffic surveillance applications

We proposed an innovative stereo camera pair calibration method suitable for traffic surveillance applications. In the method, we first split the sought parameters into two groups. The first group contains parameters that can be computed prior to the device being installed in the desired location, and we determine their values using already established methods. The second group contains parameters that need to be evaluated after the device is installed on-site, and we utilize calibration vehicles for their evaluation. We first localize the calibration vehicles by detecting and tracking their license plates in a series of stereo images; afterward, we exploit the known information about their speed and acceleration to compute the distances that they traveled between two frames; finally, we determine the internal and external stereo camera pair parameters by minimizing the difference between the computed and estimated distances while preserving the epipolar geometry. We evaluated the presented method on a task of measuring the distances that vehicles traveled between two consecutive frames. For this evaluation, we recorded a dataset containing almost 700 vehicles with trajectories that were recorded using prototype hardware and ground truth distance measurements that were obtained from a pair of single-beam LIDARs. The evaluation results were compared with the current state-of-the-art methods for long-distance stereo camera pair calibration and used in the application of the proposed method on the vehicle speed measurement task.

YPD-SLAM: A Real-Time VSLAM System for Handling Dynamic Indoor Environments.

Aiming at the problem that Simultaneous localization and mapping (SLAM) is greatly disturbed by many dynamic elements in the actual environment, this paper proposes a real-time Visual SLAM (VSLAM) algorithm to deal with a dynamic indoor environment. Firstly, a lightweight YoloFastestV2 deep learning model combined with NCNN and Mobile Neural Network (MNN) inference frameworks is used to obtain preliminary semantic information of images. The dynamic feature points are removed according to epipolar constraint and dynamic properties of objects between consecutive frames. Since reducing the number of feature points after rejection affects the pose estimation, this paper innovatively combines Cylinder and Plane Extraction (CAPE) planar detection. We generate planes from depth maps and then introduce planar and in-plane point constraints into the nonlinear optimization of SLAM. Finally, the algorithm is tested on the publicly available TUM (RGB-D) dataset, and the average improvement in localization accuracy over ORB-SLAM2, DS-SLAM, and RDMO-SLAM is about 91.95%, 27.21%, and 30.30% under dynamic sequences, respectively. The single-frame tracking time of the whole system is only 42.68 ms, which is 44.1%, being 14.6-34.33% higher than DS-SLAM, RDMO-SLAM, and RDS-SLAM respectively. The system that we proposed significantly increases processing speed, performs better in real-time, and is easily deployed on various platforms.

Single-Shot Three-Dimensional Reconstruction Using Grid Pattern-Based Structured-Light Vision Method

Structured-light vision methods are widely employed for three-dimensional reconstruction. As a typical structured light pattern, grid pattern is extensively applied in single-shot three-dimensional reconstruction. The uniqueness of the grid feature retrieval is critical to the reconstruction. Most methods using grid pattern utilize the epipolar constraint to retrieve the correspondence. However, the low calibration accuracy of the camera–projector stereo system may impact the correspondence retrieval. An approach using grid pattern-based structured-light vision method is proposed. The grid pattern-based structured-light model was combined with the camera model and the multiple light plane equations. An effective extraction method of the grid stripe features was investigated. The system calibration strategy, based on coplanar constraint, is presented. The experimental setup consisted of a camera and an LED projector. Experiments were carried out to verify the accuracy of the proposed method.

Real-time and high precision feature matching between blur aerial images.

When aerial cameras get aerial remote sensing images, the defocus will occur because of reasons such as air pressure, temperature and ground elevation changes, resulting in different image sharpness of continual aerial remote sensing images. Nowadays, the rapidly developing feature matching algorithm will rapidly reduce the registration rate between images with different image sharpness. Therefore, in order to enable aerial cameras to get image sharpness parameters according to the locations of aerial image feature points with inconsistent sharpness, this paper proposes a feature matching algorithm between aerial images with different sharpness by using DEM data and multiple constraints. In this paper, the feature matching range is extended according to the modified aerial imaging model and the nonlinear soft margin support vector machine. Then the relative moving speed and its variation of the feature points in the image are obtained by using the extended L-k optical flow, and finally the epipolar geometric constraint is introduced. To locate the feature points is obtained under multiple constraints, there is no need to calculate the feature point descriptors, and some mismatched point pairs are corrected, which improves the matching efficiency and precision. The experimental results show the feature matching precision of this algorithm is more than 90%, and the running time and matching precision can meet various application needs of aerial cameras.

YKP-SLAM: A Visual SLAM Based on Static Probability Update Strategy for Dynamic Environments

Visual simultaneous localization and mapping (SLAM) algorithms in dynamic scenes can incorrectly add moving feature points to the camera pose calculation, which leads to low accuracy and poor robustness of pose estimation. In this paper, we propose a visual SLAM algorithm based on object detection and static probability update strategy for dynamic scenes, named YKP-SLAM. Firstly, we use the YOLOv5 target detection algorithm and the improved K-means clustering algorithm to segment the image into static regions, suspicious static regions, and dynamic regions. Secondly, the static probability of feature points in each region is initialized and used as weights to solve for the initial camera pose. Then, we use the motion constraints and epipolar constraints to update the static probability of the feature points to solve the final pose of the camera. Finally, it is tested on the TUM RGB-D dataset. The results show that the YKP-SLAM algorithm proposed in this paper can effectively improve the pose estimation accuracy. Compared with the ORBSLAM2 algorithm, the absolute pose estimation accuracy is improved by 56.07% and 96.45% in low dynamic scenes and high dynamic scenes, respectively, and the best results are almost obtained compared with other advanced dynamic SLAM algorithms.

Implicit Extended Kalman Filter for Optical Terrain Relative Navigation Using Delayed Measurements

The exploration of celestial bodies such as the Moon, Mars, or even smaller ones such as comets and asteroids, is the next frontier of space exploration. One of the most interesting and attractive purposes from the scientific point of view in this field, is the capability for a spacecraft to land on such bodies. Monocular cameras are widely adopted to perform this task due to their low cost and system complexity. Nevertheless, image-based algorithms for motion estimation range across different scales of complexities and computational loads. In this paper, a method to perform relative (or local) terrain navigation using frame-to-frame features correspondences and altimeter measurements is presented. The proposed image-based approach relies on the implementation of the implicit extended Kalman filter, which works using nonlinear dynamic models and corrections from measurements that are implicit functions of the state variables. In particular, here, the epipolar constraint, which is a geometric relationship between the feature point position vectors and the camera translation vector, is employed as the implicit measurement fused with altimeter updates. In realistic applications, the image processing routines require a certain amount of time to be executed. For this reason, the presented navigation system entails a fast cycle using altimeter measurements and a slow cycle with image-based updates. Moreover, the intrinsic delay of the feature matching execution is taken into account using a modified extrapolation method.

Application of Deep Learning and Speech Recognition Technology for Pedestrian Face Recognition in health sectors

Aiming at the problems of traditional face recognition methods, this paper proposes the application of deep learning and speech recognition technology in pedestrian face recognition. Firstly, the pedestrian face image information is collected, and the face image is decomposed by wavelet scale. The improved detail enhanced face image is obtained, and Harris adaptive threshold corner detection is performed on the enhanced face image. The feature points of pedestrian face image is extracted and matched, and the local radial transformation of points and lines and the Epipolar constraint between multiple planes are adopted. Combined with the constraints of the angle and gray approximation measure of the line features of the face image, the line matching of the face close range image is completed. The 3D line features of the pedestrian face image are extracted and fitted by using the principle of face to face intersection. Combined with the pedestrian face image recognition algorithm, the pedestrian face recognition is realized. The experimental results show that the pedestrian face recognition method based on deep learning and speech recognition technology has better performance.

EPI Light Field Depth Estimation Based on a Directional Relationship Model and Multiviewpoint Attention Mechanism.

Light field (LF) image depth estimation is a critical technique for LF-related applications such as 3D reconstruction, target detection, and tracking. The refocusing property of LF images provide rich information for depth estimations; however, it is still challenging in cases of occlusion regions, edge regions, noise interference, etc. The epipolar plane image (EPI) of LF can effectively deal with the depth estimation because of its characteristics of multidirectionality and pixel consistency—in which the LF depth estimations are converted to calculate the EPI slope. This paper proposed an EPI LF depth estimation algorithm based on a directional relationship model and attention mechanism. Unlike the subaperture LF depth estimation method, the proposed method takes EPIs as input images. Specifically, a directional relationship model was used to extract direction features of the horizontal and vertical EPIs, respectively. Then, a multiviewpoint attention mechanism combining channel attention and spatial attention is used to give more weight to the EPI slope information. Subsequently, multiple residual modules are used to eliminate the redundant features that interfere with the EPI slope information—in which a small stride convolution operation is used to avoid losing key EPI slope information. The experimental results revealed that the proposed algorithm outperformed the compared algorithms in terms of accuracy.

Depth Estimation for Integral Imaging Microscopy Using a 3D–2D CNN with a Weighted Median Filter

This study proposes a robust depth map framework based on a convolutional neural network (CNN) to calculate disparities using multi-direction epipolar plane images (EPIs). A combination of three-dimensional (3D) and two-dimensional (2D) CNN-based deep learning networks is used to extract the features from each input stream separately. The 3D convolutional blocks are adapted according to the disparity of different directions of epipolar images, and 2D-CNNs are employed to minimize data loss. Finally, the multi-stream networks are merged to restore the depth information. A fully convolutional approach is scalable, which can handle any size of input and is less prone to overfitting. However, there is some noise in the direction of the edge. A weighted median filtering (WMF) is used to acquire the boundary information and improve the accuracy of the results to overcome this issue. Experimental results indicate that the suggested deep learning network architecture outperforms other architectures in terms of depth estimation accuracy.

E-Sports Training System Based on Intelligent Gesture Recognition.

In order to improve the effect of e-sports training, this paper combines the intelligent gesture recognition technology to construct an e-sports training system and judges the training effect of players through the recognition of players' gestures. Moreover, this paper studies the commonly used feature extraction algorithms and proposes an improved SLC-Harris feature extraction algorithm, and the feasibility of this algorithm is verified by the experimental results on the EUROC data set. In addition, this paper uses the KLT optical flow algorithm to track the extracted feature points and calculates the pure visual pose through epipolar geometry, triangulation, and PnP algorithms. The experimental research results show that the electronic economic training system based on intelligent gesture recognition proposed in this paper has certain effects.