Wide-baseline Stereo Research Articles

Two-View Correspondence Learning With Local Consensus Transformer.

Correspondence learning is a crucial component in multiview geometry and computer vision. The presence of heavy outliers (mismatches) consistently renders the matching problem to be highly challenging. In this article, we revisit the benefits of local consensus (LC) in traditional feature matching and introduce the concept of LC to design a trainable neural network capable of capturing the underlying correspondences. This network is named the LC transformer (LCT) and is specifically tailored for wide-baseline stereo applications. Our network architecture comprises three distinct operations. To establish the neighbor topology, we employ a dynamic graph-based embedding layer as the initial step. Subsequently, these local topologies serve as guidance for the multihead self-attention layer, enabling it to extract a more extensive contextual understanding through channel attention (CA). Following this, order-aware graph pooling is applied to extract the global context information from the embedded LC. Through the experimental analysis, the ablation study reveals that PointNet-like learning models can, indeed, benefit from the incorporation of LC. The proposed model achieves state-of-the-art performance in both challenging scenes, namely, the YFCC100M outdoor and SUN3D indoor environments, even in the presence of more than 90% outliers.

Matching wide-baseline stereo images with weak texture using the perspective invariant local feature transformer

The development of remote sensing sensor techniques allows us to now readily capture many types of indoor and outdoor scene images, which often include many weak texture regions with notable geometric distortions. Obtaining qualified matches from these difficult stereo images using existing methods is challenging. The recent achievements of deep-learning models have shown that the convolutional neural network (CNN) is adept at the image matching task. However, in practical applications, the following challenges remain: first, it is difficult to detect features in the weak texture regions of an image, and existing CNNs fail to extract discriminative image information from the quantized features of weak texture; second, as a result of the complex distortion across wide-baseline stereo images, it is difficult to match feature primitives detected in the image pair. To solve these problems, we propose the perspective invariant local feature transformer (PILFT) algorithm. Our method includes four main steps. (1) The affine scale-invariant feature transform is proposed to automatically extract the corresponding features from images, and then the perspective of the matched image is corrected to eliminate as much geometric deformation as possible. (2) The residual network is used to extract potential features from stereo images to obtain coarse and fine feature maps at different scales. (3) Using an attention mechanism, location and context information are added to the coarse level features, which are predicted by a dual-softmax function layer. (4) The features are precisely predicted on the fine feature map using the coarse reference, and the final matching results are determined by calculating the matching probability. A large number of experiments on wide-baseline weak texture images demonstrate that the proposed method has advantages over the existing algorithms in the number of matches, correct match rate, and matching accuracy. The pseudocodes of PILFT are available at https://github.com/KiltAB/PILFT.

Review of Wide-Baseline Stereo Image Matching Based on Deep Learning

Strong geometric and radiometric distortions often exist in optical wide-baseline stereo images, and some local regions can include surface discontinuities and occlusions. Digital photogrammetry and computer vision researchers have focused on automatic matching for such images. Deep convolutional neural networks, which can express high-level features and their correlation, have received increasing attention for the task of wide-baseline image matching, and learning-based methods have the potential to surpass methods based on handcrafted features. Therefore, we focus on the dynamic study of wide-baseline image matching and review the main approaches of learning-based feature detection, description, and end-to-end image matching. Moreover, we summarize the current representative research using stepwise inspection and dissection. We present the results of comprehensive experiments on actual wide-baseline stereo images, which we use to contrast and discuss the advantages and disadvantages of several state-of-the-art deep-learning algorithms. Finally, we conclude with a description of the state-of-the-art methods and forecast developing trends with unresolved challenges, providing a guide for future work.

DSEC: A Stereo Event Camera Dataset for Driving Scenarios

Once an academic venture, autonomous driving has received unparalleled corporate funding in the last decade. Still, operating conditions of current autonomous cars are mostly restricted to ideal scenarios. This means that driving in challenging illumination conditions such as night, sunrise, and sunset remains an open problem. In these cases, standard cameras are being pushed to their limits in terms of low light and high dynamic range performance. To address these challenges, we propose, DSEC, a new dataset that contains such demanding illumination conditions and provides a rich set of sensory data. DSEC offers data from a wide-baseline stereo setup of two color frame cameras and two high-resolution monochrome event cameras. In addition, we collect lidar data and RTK GPS measurements, both hardware synchronized with all camera data. One of the distinctive features of this dataset is the inclusion of high-resolution event cameras. Event cameras have received increasing attention for their high temporal resolution and high dynamic range performance. However, due to their novelty, event camera datasets in driving scenarios are rare. This work presents the first high resolution, large scale stereo dataset with event cameras. The dataset contains 53 sequences collected by driving in a variety of illumination conditions and provides ground truth disparity for the development and evaluation of event-based stereo algorithms.

Efficient Deterministic Search With Robust Loss Functions for Geometric Model Fitting.

Geometric model fitting is a fundamental task in computer vision, which serves as the pre-requisite of many downstream applications. While the problem has a simple intrinsic structure where the solution can be parameterized within a few degrees of freedom, the ubiquitously existing outliers are the main challenge. In previous studies, random sampling techniques have been established as the practical choice, since optimization-based methods are usually too time-demanding. This prospective study is intended to design efficient algorithms that benefit from a general optimization-based view. In particular, two important types of loss functions are discussed, i.e., truncated and l1 losses, and efficient solvers have been derived for both upon specific approximations. Based on this philosophy, a class of algorithms are introduced to perform deterministic search for the inliers or geometric model. Recommendations are made based on theoretical and experimental analyses. Compared with the existing solutions, the proposed methods are both simple in computation and robust to outliers. Extensive experiments are conducted on publicly available datasets for geometric estimation, which demonstrate the superiority of our methods compared with the state-of-the-art ones. Additionally, we apply our method to the recent benchmark for wide-baseline stereo evaluation, leading to a significant improvement of performance. Our code is publicly available at https://github.com/AoxiangFan/EifficientDeterministicSearch.

Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems.

Many classical Computer Vision problems, such as essential matrix computation and pose estimation from 3D to 2D correspondences, can be tackled by solving a linear least-square problem, which can be done by finding the eigenvector corresponding to the smallest, or zero, eigenvalue of a matrix representing a linear system. Incorporating this in deep learning frameworks would allow us to explicitly encode known notions of geometry, instead of having the network implicitly learn them from data. However, performing eigendecomposition within a network requires the ability to differentiate this operation. While theoretically doable, this introduces numerical instability in the optimization process in practice. In this paper, we introduce an eigendecomposition-free approach to training a deep network whose loss depends on the eigenvector corresponding to a zero eigenvalue of a matrix predicted by the network. We demonstrate that our approach is much more robust than explicit differentiation of the eigendecomposition using two general tasks, outlier rejection and denoising, with several practical examples including wide-baseline stereo, the perspective-n-point problem, and ellipse fitting. Empirically, our method has better convergence properties and yields state-of-the-art results.

Real-time 6D Racket Pose Estimation and Classification for Table Tennis Robots

For table tennis robots, it is a significant challenge to understand the opponent's movements and return the ball accordingly with high performance. One has to cope with various ball speeds and spins resulting from different stroke types. In this paper, we propose a real-time 6D racket pose detection method and classify racket movements into five stroke categories with a neural network. By using two monocular cameras, we can extract the racket's contours and choose some special points as feature points in image coordinates. With the 3D geometrical information of a racket, a wide baseline stereo matching method is proposed to find the corresponding feature points and compute the 3D position and orientation of the racket by triangulation and plane fitting. Then, a Kalman filter is adopted to track the racket pose, and a multilayer perceptron (MLP) neural network is used to classify the pose movements. We conduct two experiments to evaluate the accuracy of racket pose detection and classification, in which the average error in position and orientation is around 7.8 mm and 7.2 by comparing with the ground truth from a KUKA robot. The classification accuracy is 98%, the same as the human pose estimation method with Convolutional Pose Machines (CPMs).

Efficient cost aggregation for feature-vector-based wide-baseline stereo matching

■■■In stereo matching applications, local cost aggregation techniques are usually preferred over global methods due to their speed and ease of implementation. Local methods make implicit smoothness assumptions by aggregating costs within a finite window; however, cost aggregation is a time-consuming process. Furthermore, most existing local methods are based on pixel intensity values, and hence are not efficient with feature vectors used in wide-baseline stereo matching. In this paper, a new cost aggregation method is proposed, where a Per-Column Cost matrix is combined with a feature-vector-based weighting strategy to achieve both matching accuracy and computational efficiency. Here, the proposed cost aggregation method is applied with the DAISY feature descriptor for wide-baseline stereo matching; however, this method can also be applied to a fast growing number of stereo matching techniques that are based on feature descriptors. A performance comparison with several benchmark local cost aggregation approaches is presented, along with a thorough analysis of the time and storage complexity of the proposed method.

Robust Harris Corner Matching Based on the Quasi-Homography Transform and Self-Adaptive Window for Wide-Baseline Stereo Images

Corner feature matching remains a difficult task for wide-baseline images because of viewpoint distortion, surface discontinuities, and partial occlusions. In this paper, we propose a robust Harris corner matching method based on the quasi-homography transform (QHT) and self-adaptive window. Our method is divided into three steps. First, high-quality Harris corners were extracted from stereo images using optimal detecting, and initial matches were simultaneously acquired by integrating complementary affine-invariant features and the scale-invariant feature transform descriptor. Second, the pair of fundamental matrices was estimated based on the initial matches and improved random sample consensus algorithm. Subsequently, the global QHT was produced by duplicate epipolar geometries. Third, conjugate Harris corners were obtained by combining QHT and normalized cross correlation, and the accuracy of the corresponding points was further improved based on self-adaptive least-squares matching (SALSM). Experiments on six groups of wide-baseline images demonstrate the effectiveness of the proposed method, and a comprehensive comparison with the existing corner matching algorithms indicates that our method has notable superiority in terms of accuracy and distribution. The main contribution of this paper is that the proposed global QHT can reduce the search range effectively for candidates, and the proposed SALSM can notably improve the accuracy of the corresponding corners.

Vision-based navigation of an unmanned surface vehicle with object detection and tracking abilities

The paper discusses autocalibration, object detection, and object tracking for unmanned surface vehicles. Input data are recorded with a wide-baseline stereo vision system providing accuracy for distance estimations. The paper reports about followed ways and novel contributions for ensuring a working system solution. Automatic self-calibration is used for the wide-baseline stereo vision system. Robust sea surface estimation and the detection of the horizon support the understanding of the given scene environment. Long-range (i.e. up to 500 m) object detection and tracking are supported by the used wide-baseline stereo system. The paper informs about the complete system design, informs about applied or designed methods, and also about experiments which verify that the system achieved an operational state.

EasyFlow: increasing the convergence basin of variational image matching with a feature‐based cost

Dense motion field estimation is a key computer vision problem. Many solutions have been proposed to compute small or large displacements, narrow or wide baseline stereo disparity, or non‐rigid surface registration, but a unified methodology is still lacking. The authors introduce a general framework that robustly combines direct and feature‐based matching. The feature‐based cost is built around a novel robust distance function that handles keypoints and weak features such as segments. It allows us to use putative feature matches to guide dense motion estimation out of local minima. The authors’ framework uses a robust direct data term. It is implemented with a powerful second‐order regularisation with external and self‐occlusion reasoning. Their framework achieves state‐of‐the‐art performance in several cases (standard optical flow benchmarks, wide‐baseline stereo and non‐rigid surface registration). Their framework has a modular design that customises to specific application needs.

ENGLISH

The objective here is to present a briefing on the methods and algorithms presented in the field of identifiers and descriptors of key point areas and the manner of obtaining various indicators in order to comply and track the selected objects, compare and reveal their advantages and disadvantages in 2D features in recent decades. Feature detection methods subject to study in this article consist of:Scale-Invariant Feature; Maximally Stable Extremal Regions; Speeded up Robust Features; Features from Accelerated Segment Test; Binary Robust Independent Elementary Features; Efficient Dense Descriptor Applied to Wide-Baseline Stereo; Binary Robust Invariant Scalable Key points; Oriented Binary Robust Independent Elementary Features and Fast Retina Key point.

Automated localisation of Mars rovers using co-registered HiRISE-CTX-HRSC orthorectified images and wide baseline Navcam orthorectified mosaics

We present a wide range of research results in the area of orbit-to-orbit and orbit-to-ground data fusion, achieved within the EU-FP7 PRoVisG project and EU-FP7 PRoViDE project. We focus on examples from three Mars rover missions, i.e. MER-A/B and MSL, to provide examples of a new fully automated offline method for rover localisation. We start by introducing the mis-registration discovered between the current HRSC and HiRISE datasets. Then we introduce the HRSC to CTX and CTX to HiRISE co-registration workflow. Finally, we demonstrate results of wide baseline stereo reconstruction with fixed mast position rover stereo imagery and its application to ground-to-orbit co-registration with HiRISE orthorectified image. We show examples of the quantitative assessment of recomputed rover traverses, and extensional exploitation of the co-registered datasets in visualisation and within an interactive web-GIS.

Stereo fusion: Combining refractive and binocular disparity

The performance of depth reconstruction in binocular stereo relies on how adequate the predefined baseline for a target scene is. Wide-baseline stereo is capable of discriminating depth better than the narrow-baseline stereo, but it often suffers from spatial artifacts. Narrow-baseline stereo can provide a more elaborate depth map with fewer artifacts, while its depth resolution tends to be biased or coarse due to the short disparity. In this paper, we propose a novel optical design of heterogeneous stereo fusion on a binocular imaging system with a refractive medium, where the binocular stereo part operates as wide-baseline stereo, and the refractive stereo module works as narrow-baseline stereo. We then introduce a stereo fusion workflow that combines the refractive and binocular stereo algorithms to estimate fine depth information through this fusion design. In addition, we propose an efficient calibration method for refractive stereo. The quantitative and qualitative results validate the performance of our stereo fusion system in measuring depth in comparison with homogeneous stereo approaches.

The Bubble Box: Towards an Automated Visual Sensor for 3D Analysis and Characterization of Marine Gas Release Sites.

Several acoustic and optical techniques have been used for characterizing natural and anthropogenic gas leaks (carbon dioxide, methane) from the ocean floor. Here, single-camera based methods for bubble stream observation have become an important tool, as they help estimating flux and bubble sizes under certain assumptions. However, they record only a projection of a bubble into the camera and therefore cannot capture the full 3D shape, which is particularly important for larger, non-spherical bubbles. The unknown distance of the bubble to the camera (making it appear larger or smaller than expected) as well as refraction at the camera interface introduce extra uncertainties. In this article, we introduce our wide baseline stereo-camera deep-sea sensor bubble box that overcomes these limitations, as it observes bubbles from two orthogonal directions using calibrated cameras. Besides the setup and the hardware of the system, we discuss appropriate calibration and the different automated processing steps deblurring, detection, tracking, and 3D fitting that are crucial to arrive at a 3D ellipsoidal shape and rise speed of each bubble. The obtained values for single bubbles can be aggregated into statistical bubble size distributions or fluxes for extrapolation based on diffusion and dissolution models and large scale acoustic surveys. We demonstrate and evaluate the wide baseline stereo measurement model using a controlled test setup with ground truth information.

Wide baseline stereo matching based on scale invariant feature transformation with hybrid geometric constraints

Wide baseline stereo matching is a challenging task because of the presence of significant geometric deformations and illumination changes within the images. Based on the scale invariant feature transformation (SIFT) algorithm, this study proposes a new hybrid matching scheme that uses both the feature-based and the area-based methods to find reliable matches from sparse to dense under different geometric constraints. Firstly, the authors propose a SIFT-based robust weighted least squares matching (LSM) method modelled by a two-dimensional (2D) projective transformation to establish the initial correspondences and their local homographies. In this method, a normalised cross correlation metric modified with an adaptive scale and an orientation of the SIFT features (SIFT-NCC) is proposed to find a good initial alignment for the SIFT-LSM. Secondly, a robust matching propagation using the SIFT-NCC starts from the initial matches under an epipolar geometry and the local homography constraints; geometrical consistency checking is used simultaneously to identify the false matches. Thirdly, they use an improved, feature-based SIFT matching method to find the correspondences from the points that are not coplanar in the 3D space under an epipolar constraint only. A bidirectional selection strategy is used to remove the error matches.

Improving Video Stabilization Using Multi-Resolution MSER Features

ABSTRACTIn this paper, we investigate the application of multi-resolution maximally stable extremal region (MSER) features for improving the video stabilization performance. MSER features have been used for many computer vision applications like wide baseline stereo, object recognition, video object tracking, and video stabilization with very good results as compared to other features like scale invariant feature transform (SIFT) and Kanade Lucas Tomasi (KLT). However, a limitation of the MSER feature in the stabilization application was observed when the input video frames were severely blurred. The same limitation was also observed when other features like KLT and SIFT were utilized under blurring conditions. In this paper we propose to overcome this drawback for video stabilization application by utilizing MSERs which are extracted and matched in a scale pyramid fashion instead of the MSER features detected and matched on a single image resolution. The duplicate MSERs resulting due to the pyramid style detection are removed followed by MSER feature matching for establishing correspondence between video frames to estimate the global motion parameters. Once the global motion parameters are estimated, the accumulated transformation is smoothed followed by motion compensation to construct the stabilized frame. Comparative analysis with state-of-the-art stabilization methods shows improvement in stabilization performance as well as robustness to blurring degradations. The proposed method can easily be ported to other feature detectors like KLT and SIFT thereby making the proposed method generic to any feature detector.

Exploiting local linear geometric structure for identifying correct matches

Selecting correct matches from a set of tentative feature point correspondences plays a vital important role in many tasks, such as structure from motion (SfM), wide baseline stereo and image search. In this paper, we propose an efficient and effective method for identifying correct matches from an initial batch of feature correspondences. The proposed method first obtains a subset of correct matches based on the assumption that the local geometric structure among a feature point and its nearest neighbors in an image cannot be easily affected by both geometric and photometric transformations, and thus should be observed in the matched images. For efficiency, we model this local geometric structure by a set of linear coefficients that reconstruct the point from its neighbors. After obtaining a portion of correct matches, we then provide two ways to accurately estimate the correctness of each match and to efficiently estimate the number of correct matches, respectively. The proposed method is evaluated on both applications including image matching and image re-ranking. Experimental results on several public datasets show that our method outperforms state-of-the-art techniques in terms of speed and accuracy.

3D Data Products and Web-GIS for Mars Rover Mission for Seamless Visualisation from Orbit to Ground-level

Abstract. This paper presents a wide range of research and processing results in the area of multi-resolution orbital data co-registration, multiresolution ground 3D reconstruction, and orbit-to-ground data fusion, achieved within the EU-FP7 PRoVisG and PRoViDE project. We focus on three NASA rover missions, MER-A, MER-B, and MSL, to provide examples of automated methods for producing coregistered, multi-resolution 3D products. We highlight the mis-registration discovered between current HiRISE to HRSC datasets, CTX to HRSC and HiRISE to CTX co- results, wide baseline stereo reconstruction results of rover imagery, ground-to-orbit coregistration, i.e. reconstructed wide baseline ground ORI and HiRISE ORI co-registration, and extensive exploitation of the coregistered datasets in visualisation and interactive web-GIS.

Wide-baseline stereo matching based on the line intersection context for real-time workspace modeling

Line matching in widely separated views is challenging because of large perspective distortion and violation of the planarity assumption in local regions. We introduce a novel method of wide-baseline image matching based on the coplanar line intersections for poorly textured and/or nonplanar structured scenes. The local areas of the coplanar line pairs are normalized into canonical frames by rectifying the coplanar line pairs to be orthogonal. Then, the 3D interpretation of the intersection context of the coplanar line pairs helps to match the nonplanar local regions. Furthermore, for calibrated stereo cameras, we propose a matching criterion based on 3D planar homography to improve the matching accuracy while reconstructing most likely physically existing planar patches. Experimental results demonstrate the effectiveness of the proposed method for real-world scenes.