Line Segment Matching Research Articles

A Generalized Voronoi Diagram-Based Segment-Point Cyclic Line Segment Matching Method for Stereo Satellite Images

Matched line segments are crucial geometric elements for reconstructing the desired 3D structure in stereo satellite imagery, owing to their advantages in spatial representation, complex shape description, and geometric computation. However, existing line segment matching (LSM) methods face significant challenges in effectively addressing co-linear interference and the misdirection of parallel line segments. To address these issues, this study proposes a “continuous–discrete–continuous” cyclic LSM method, based on the Voronoi diagram, for stereo satellite images. Initially, to compute the discrete line-point matching rate, line segments are discretized using the Bresenham algorithm, and the pyramid histogram of visual words (PHOW) feature is assigned to the line segment points which are detected using the line segment detector (LSD). Next, to obtain continuous matched line segments, the method combines the line segment crossing angle rate with the line-point matching rate, utilizing a soft voting classifier. Finally, local point-line homography models are constructed based on the Voronoi diagram, filtering out misdirected parallel line segments and yielding the final matched line segments. Extensive experiments on the challenging benchmark, WorldView-2 and WorldView-3 satellite image datasets, demonstrate that the proposed method outperforms several state-of-the-art LSM methods. Specifically, the proposed method achieves F1-scores that are 6.22%, 12.60%, and 18.35% higher than those of the best-performing existing LSM method on the three datasets, respectively.

The One-Point-One-Line geometry for robust and efficient line segment correspondence

Three-dimensional (3D) lines are common elements in artificial scenes and serve as basic, yet essential features for structural 3D reconstruction. The crucial step of 3D line reconstruction, namely two-view line segment matching, still faces challenges in terms of both accuracy and efficiency improvements. Therefore, robust and efficient constraints are needed to establish valid line candidates. This paper introduces a novel geometry constraint called “one-point-one-line geometry” (OPOL) to enhance the precision of line matching and reduce computational complexity. OPOL offers two remarkable advantages: (1) It takes point orientations as the constraint, which is not only invariant to projective transformations, but also alleviates computational requirements. (2) It needs only one point match to construct the geometry constraint, thus both the grouping and validation are greatly reduced. Additionally, we incorporate the line sweep strategy into OPOL, leveraging depth and space constraints derived from existing 3D points to further enhance efficiency. Extensive experiments conducted on large-coverage and high-resolution images (as large as 10336 × 7788 pixels) demonstrated that OPOL matched lines within a second for an image pair. Both quantitative and qualitative results also demonstrated the superior accuracy and efficiency performance of OPOL. We integrated OPOL into multiple view line reconstruction frameworks, and the promising experimental results reveal the performance of OPOL for robust line reconstruction. The OPOL code is publicly available at https://github.com/JoeAlexxxxx/OPOL.

Hierarchical Edge-Preserving Dense Matching by Exploiting Reliably Matched Line Segments

Image dense matching plays a crucial role in the reconstruction of three-dimensional models of buildings. However, large variations in target heights and serious occlusion lead to obvious mismatches in areas with discontinuous depths, such as building edges. To solve this problem, the present study mines the geometric and semantic information of line segments to produce a constraint for the dense matching process. First, a disparity consistency-based line segment matching method is proposed. This method correctly matches line segments on building structures in discontinuous areas based on the assumption that, within the corresponding local areas formed by two corresponding line pairs, the disparity obtained by the coarse-level matching of the hierarchical dense matching is similar to that derived from the local homography estimated from the corresponding line pairs. Second, an adaptive guide parameter is designed to constrain the cost propagation between pixels in the neighborhood of line segments. This improves the rationality of cost aggregation paths in discontinuous areas, thereby enhancing the matching accuracy near building edges. Experimental results using satellite and aerial images show that the proposed method efficiently obtains reliable line segment matches at building edges with a matching precision exceeding 97%. Under the constraint of the matched line segments, the proposed dense matching method generates building edges that are visually clearer, and achieves higher accuracy around edges, than without the line segment constraint.

Robust line segment mismatch removal using point-pair representation and Gaussian-uniform mixture formulation

Line segment matching (LSM) plays an important role in image matching, while it is always a challenging problem due to line fracture and high geometric complexity. In this paper, we propose a line segment mismatch removal to address the sensitivity to segment length and the fracture problem. The mismatch removal removes the massive false matches obtained by the descriptors like LBD. Specifically, we suggest calculating the endpoint error in LSM rather than the direction error in considering the error band and segment length. We propose a point-pair representation, and transform the LSM problem to be an intuitive point-pair alignment problem with a mixture model. The point-pair representation is more robust on the segment length than the linear equation representation used in RANSAC, which significantly preserves short line segments. Then, based on the point-pair representation, we propose a novel Directed Endpoint Drift method to solve the inherent fracture problem of line segments by allowing the endpoints to move along the line to complement the fracture. Compared with the recent learning-based method, the proposed method nearly doubles the number of correct matches on the remote sensing dataset. Especially, the recall is improved by 10% to 30% for both the short segments and the slightly fractured segments, compared with the popular RANSAC and MAGSAC++ methods. The code is available at https://github.com/shenliang16/DEpD.

PLDS-SLAM: Point and Line Features SLAM in Dynamic Environment

Visual simultaneous localization and mapping (SLAM), based on point features, achieves high localization accuracy and map construction. They primarily perform simultaneous localization and mapping based on static features. Despite their efficiency and high precision, they are prone to instability and even failure in complex environments. In a dynamic environment, it is easy to keep track of failures and even failures in work. The dynamic object elimination method, based on semantic segmentation, often recognizes dynamic objects and static objects without distinction. If there are many semantic segmentation objects or the distribution of segmentation objects is uneven in the camera view, this may result in feature offset and deficiency for map matching and motion tracking, which will lead to problems, such as reduced system accuracy, tracking failure, and track loss. To address these issues, we propose a novel point-line SLAM system based on dynamic environments. The method we propose obtains the prior dynamic region features by detecting and segmenting the dynamic region. It realizes the separation of dynamic and static objects by proposing a geometric constraint method for matching line segments, combined with the epipolar constraint method of feature points. Additionally, a dynamic feature tracking method based on Bayesian theory is proposed to eliminate the dynamic noise of points and lines and improve the robustness and accuracy of the SLAM system. We have performed extensive experiments on the KITTI and HPatches datasets to verify these claims. The experimental results show that our proposed method has excellent performance in dynamic and complex scenes.

High Precision Mesh-Based Drone Image Stitching Based on Salient Structure Preservation and Regular Boundaries

Addressing problems such as obvious ghost, dislocation, and distortion resulting from the traditional stitching method, a novel drone image-stitching method is proposed using mesh-based local double-feature bundle adjustment and salient structure preservation which aims to obtain more natural panoramas.The proposed method is divided into the following steps. First, reducing parallax error is considered from both global and local aspects. Global bundle adjustment is introduced to minimize global transfer error, and then the local mesh-based feature-alignment model is incorporated into the optimization framework to achieve more accurate alignment. Considering the sensitivity of human eyes to linear structure, the global linear structure that runs through the images obtained by segment fusion is introduced to prevent distortions and align matching line segments better. Rectangular panoramas usually have better visual effects. Therefore, regular boundary constraint combined with mesh-based shape-preserving transform can make the results more natural while preserving mesh geometry. Two new evaluation metrics are also developed to quantify the performance of linear structure preservation and the alignment difference of matching line segments. Extensive experiments show that our proposed method can eliminate parallax and preserve global linear structures better than other state-of-the-art stitching methods and obtain more natural-looking stitching results.

An adaptive feature region-based line segment matching method for viewpoint-changed images with discontinuous parallax and poor textures

Many methods have been proposed to extract line segment (LS) correspondences for images with viewpoint variations. However, the matching performance on images with viewpoint variations is still limited due to the influence of discontinuous parallax and poor textures on images. Existing line segment matching (LSM) methods rarely consider the coexistence of these two situations. This study attempts to address this problem by proposing an adaptive feature region (FR)-based LSM method based on the observation that different FRs are suitable for different image scenes under image viewpoint change. In the proposed method, LSs are paired, and two types of FRs, symmetric FR centered on the line-pair intersection and asymmetric FR with the intersection as a vertex, are constructed for every line pair. The asymmetric FR of a line pair is a subregion of its symmetric FR. And the complement region of asymmetric FR in symmetric FR is called complement FR. To choose a suitable FR for the matching of each line pair adaptively, the asymmetric FR-based feature descriptor similarity and epipolar line-based constraints are combined to determine candidate line pair matches firstly. Then, the candidate matches with similar complement FR-based feature descriptors are considered initial matches and used to construct a topological constraint to check other candidates. Finally, we estimate an adaptive influence region-based local homography for each matched line pair to constrain the matching of unmatched individual LSs. The experimental results on the test images with viewpoint changes show that our approach outperforms the state-of-the-art methods in terms of Recall, matching precision (MP), and F-Measure (reflecting the overall matching performance of Recall and MP). Especially, it improves the average Recall and average F-Measure of the best method among the comparison methods by 45.20% and 25.83%, respectively.

Advanced-Technological UAVs-Based Enhanced Reconstruction of Edges for Building Models

Accurate building models are widely used in the construction industry in the digital era. UAV cameras combined with image-based reconstruction provide a low-cost technology for building modeling. Most existing reconstruction methods operate on point clouds, while massive points reduce computational efficiency, and the accumulated error of point position often distorts building edges. This paper introduces an innovative 3D reconstruction method, Edge3D, that recovers building edges in the form of 3D lines. It employs geometry constraints and progressive screening technology to improve the robustness and precision of line segment matching. An innovative bundle adjustment strategy based on endpoints is designed to reduce the global reprojection error. Edges were tested on challenging real-world image sets, and matching precisions of 96% and 94% were achieved on the two image sets, respectively, with good reconstruction results. The proposed approach reconstructs building edges using a small number of lines instead of massive points, which contributes to the rapid reconstruction of building contour construction and obtaining accurate models, serving as an important foundation for the promotion of construction advancement.

A Novel Multispectral Line Segment Matching Method Based on Phase Congruency and Multiple Local Homographies

Feature matching is a fundamental procedure in several image processing methods applied in remote sensing. Multispectral sensors with different wavelengths can provide complementary information. In this work, we propose a multispectral line segment matching algorithm based on phase congruency and multiple local homographies (PC-MLH) for image pairs captured by the cross-spectrum sensors (visible spectrum and infrared spectrum) in man-made scenarios. The feature points are first extracted and matched according to phase congruency. Next, multi-layer local homographies are derived from clustered feature points via random sample consensus (RANSAC) to guide line segment matching. Moreover, three geometric constraints (line position encoding, overlap ratio, and point-to-line distance) are introduced in cascade to reduce the computational complexity. The two main contributions of our work are as follows: First, compared with the conventional line matching methods designed for single-spectrum images, PC-MLH is robust against nonlinear radiation distortion (NRD) and can handle the unknown multiple local mapping, two common challenges associated with multispectral feature matching. Second, fusion of line extraction results and line position encoding for neighbouring matching increase the number of matched line segments and speed up the matching process, respectively. The method is validated using two public datasets, CVC-multimodal and VIS-IR. The results show that the percentage of correct matches (PCM) using PC-MLH can reach 94%, which significantly outperforms other single-spectral and multispectral line segment matching methods.

Towards Light-Weight and Real-Time Line Segment Detection

Previous deep learning-based line segment detection (LSD) suffers from the immense model size and high computational cost for line prediction. This constrains them from real-time inference on computationally restricted environments. In this paper, we propose a real-time and light-weight line segment detector for resource-constrained environments named Mobile LSD (M-LSD). We design an extremely efficient LSD architecture by minimizing the backbone network and removing the typical multi-module process for line prediction found in previous methods. To maintain competitive performance with a light-weight network, we present novel training schemes: Segments of Line segment (SoL) augmentation, matching and geometric loss. SoL augmentation splits a line segment into multiple subparts, which are used to provide auxiliary line data during the training process. Moreover, the matching and geometric loss allow a model to capture additional geometric cues. Compared with TP-LSD-Lite, previously the best real-time LSD method, our model (M-LSD-tiny) achieves competitive performance with 2.5% of model size and an increase of 130.5% in inference speed on GPU. Furthermore, our model runs at 56.8 FPS and 48.6 FPS on the latest Android and iPhone mobile devices, respectively. To the best of our knowledge, this is the first real-time deep LSD available on mobile devices.

A Visual Compass Based on Point and Line Features for UAV High-Altitude Orientation Estimation

The accurate and reliable high-altitude orientation estimation is of great significance for unmanned aerial vehicles (UAVs) localization, and further assists them to conduct some fundamental functions, such as aerial mapping, environmental monitoring, and risk management. However, the traditional orientation estimation is susceptible to electromagnetic interference, high maneuverability, and substantial scale variations. Hence, this paper aims to present a new visual compass algorithm to estimate the orientation of a UAV employing the appearance and geometry structure of the point and line features in the remote sensing images. In this study, a coarse-to-fine feature tracking method is used to locate the matched keypoints precisely. An LK-ZNCC algorithm is proposed to match line segments in real-time. A hierarchical fusion method for point and line features is designed to expand the scope of the usage of this system. Many comparative experiments between this algorithm and others are conducted on a UAV. Experimental results show that the proposed visual compass algorithm is a reliable, precise, and versatile system applicable to other UAV navigation systems, especially when they do not work in particular situations.

Line Segment Matching Fusing Local Gradient Order and Non-Local Structure Information

Line segment matching is essential for industrial applications such as scene reconstruction, pattern recognition, and VSLAM. To achieve good performance under the scene with illumination changes, we propose a line segment matching method fusing local gradient order and non-local structure information. This method begins with intensity histogram multiple averaging being utilized for adaptive partitioning. After that, the line support region is divided into several sub-regions, and the whole image is divided into a few intervals. Then the sub-regions are encoded by local gradient order, and the intervals are encoded by non-local structure information of the relationship between the sampled points and the anchor points. Finally, two histograms of the encoded vectors are, respectively, normalized and cascaded. The proposed method was tested on the public datasets and compared with previous methods, which are the line-junction-line (LJL), the mean-standard deviation line descriptor (MSLD) and the line-point invariant (LPI). Experiments show that our approach has better performance than the representative methods in various scenes. Therefore, a tentative conclusion can be drawn that this method is robust and suitable for various illumination changes scenes.

Smoothly varying projective transformation for line segment matching

Line segment matching is important in applications that require recovering the 3D structure of objects (e.g., manmade objects in street-level scenarios). However, differentiating between true and false line matches is generally difficult without strong geometric constraints for line segments. Hence, additional constraints are forced to be used, sacrificing many true line matches. This study proposes a robust line segment matching method based on global projective transformation modeling. We develop a non-parametric motion regression formulation with a specially designed direct linear transformation-based cost function that reformulates the piecewise smoothly varying projective transformations as a global continuous model from highly noisy point matches. The resultant model can effectively approximate the real underlying image transformation and derive high-quality point matches. We apply the computed model and high-quality point matches to a point-correspondence-based line matching pipeline, which provides sufficient strict geometric constraints for first generating the pair-to-pair matches and then distilling the line-to-line matches. Extensive experiments conducted on two challenging line matching datasets show that the proposed method can obtain considerable correct line segment matches, outperforming the comparison methods in mean F-score by at least 15.5% on the benchmark dataset and 16.9% on the local dataset. Code is available at https://github.com/geovsion/SLEM.

Hierarchical line segment matching for wide-baseline images via exploiting viewpoint robust local structure and geometric constraints

Line segment matching for wide-baseline images is challenging due to the significant viewpoint differences. In this study, we propose a hierarchical line segment matching method based on viewpoint robust local structure and geometric constraints. In our approach, line segments are paired and classified into three types representing heuristically those with different level of matchability: structured line pairs (S-LPs), unstructured line pairs (U-LPs), and individual line segments (I-LSs). Accordingly, we design a hierarchical matching framework that consists of three matching layers respectively corresponding to the above three types: in the first layer, robust local structures are constructed for S-LPs. We match the S-LPs by measuring local structure similarity (LSS). In the second layer, we build a topological descriptor-based constraint based on the S-LP matches and combine it with epipolar geometry-based constraints to select candidate matches for U-LPs, and then use LSS to further refine the matches. In the third layer, we estimate local homography for I-LSs to build constraints, to extract matches by exploiting the implicit region information of line pair matches. A pair of pre- and postprocessing algorithms, namely line segment merging and match reassignment, are performed before and after the matching procedure to overcome the negative effect of line segment fragmentation on the matching. Experimental results demonstrate that the proposed method performs better than state-of-the-art methods (with the largest improvement of 124.17% in terms of F-Measure over the best one among the compared methods).

A Monocular Visual Odometry Method Based on Virtual-Real Hybrid Map in Low-Texture Outdoor Environment.

With the extensive application of robots, such as unmanned aerial vehicle (UAV) in exploring unknown environments, visual odometry (VO) algorithms have played an increasingly important role. The environments are diverse, not always textured, or low-textured with insufficient features, making them challenging for mainstream VO. However, for low-texture environment, due to the structural characteristics of man-made scene, the lines are usually abundant. In this paper, we propose a virtual-real hybrid map based monocular visual odometry algorithm. The core idea is that we reprocess line segment features to generate the virtual intersection matching points, which can be used to build the virtual map. Introducing virtual map can improve the stability of the visual odometry algorithm in low-texture environment. Specifically, we first combine unparallel matched line segments to generate virtual intersection matching points, then, based on the virtual intersection matching points, we triangulate to get a virtual map, combined with the real map built upon the ordinary point features to form a virtual-real hybrid 3D map. Finally, using the hybrid map, the continuous camera pose estimation can be solved. Extensive experimental results have demonstrated the robustness and effectiveness of the proposed method in various low-texture scenes.

WGLSM: An end-to-end line matching network based on graph convolution

Line matching plays an essential role in Structure from Motion (SFM) and Simultaneous Localization and Mapping (SLAM), especially in low-texture scenes, where feature points are hard to be detected. In this paper, we present a new method by combining Convolutional Neural Networks and Graph Convolutional Networks to match line segments in pairs of images. We design a graph-based method to predict the assignment matrix of two feature sets with solving a relaxed optimal transport problem. In contrast to handcrafted line matching algorithms, our approach learns the line segment features and performs matching simultaneously through end-to-end weakly supervised training. The experiment results show that our method outperforms the state-of-the-art techniques and is robust to various image transformations. Besides, the generalization experiment illustrates that our method has good generalization ability without fine-tuning. The code of our work is available athttps://github.com/mameng1/GraphLineMatching.

Real time Localization Method Research with Monoslam and BIM Model

Outdoor navigation technologies based on GPS and Beidou has been very mature. However, due to the weakness of satellite signal and refraction, they can’t satisfy the needs of indoor localization and parking area navigation in large commercial buildings. Most indoor localization technologies, such as Bluetooth and Wi-Fi, are based on active equipment. Due to the implementation cost, they can’t be widely deployed. In recent years, SLAM has been greatly developed and applied in the field of robotics and autonomous driving. For technologies such as stereo SLAM and Lidar SLAM require additional equipment, and meanwhile large scale computing will happen during point cloud matching, they are not suitable for civilian mobile usage. Considering the BIM technology, which can provide accurate 3D data of buildings, are widely used gradually, a real-time localization method combining monocular SLAM and BIM model is proposed to complete vision-based initial localization. First, we create the BIM model with the design drawings, extract the visible edge lines of model, and establish a 3D line feature database; then, intercept the key frames of the mobile camera, and extract the edge line and line feature descriptors from image, match the line features between key frames, triangulate the matched line segments, get the 3D line coordinate in first frame, mistake filter will be rolled out between multiple frames; finally, match the reconstruction result with line feature database form BIM model, register the initial position. This method can quickly and effectively perform the registration of monocular images and BIM models, get the initial position in complex spaces, and support the application scenarios such as indoor navigation, equipment maintenance, facility management, and construction monitoring.

Robust line feature matching based on pair-wise geometric constraints and matching redundancy

This paper presents a novel method for matching line segments in images based on pair-wise geometric constraints and matching redundancy. In this study, pairs of line segments satisfying angle and distance constraints are used as matching primitives. To ensure that each extracted line segment is paired with another line segment, the search region of each line segment is gradually grown until it is paired. Initial pair-to-pair correspondences between two images are established using four pair-wise constraints; next, line-to-line correspondences are obtained. To effectively solve the matching conflict in the results, a method of recording the result of line pair matching based on a double-layer matrix is proposed. Based on the double-layer matrix, an effective checking method for the line matching results based on the collinearity constraint and matching redundancy is presented. It fully utilizes redundancy information and considers the collinearity of fragmented line segments. Further, it can effectively separate correct and incorrect matches from the one-to-many, many-to-one, and many-to-many matching results. The proposed method was tested on 12 image pairs from a benchmark of matched-lines, and compared with other state-of-the-art methods. The results demonstrate the superiority of the proposed method due to its higher accuracy and greater recall in challenging cases.

Robust line segment matching across views via ranking the line-point graph

Line segment matching in two or multiple views is helpful to 3D reconstruction and pattern recognition. To fully utilize the geometry constraint of different features for line segment matching, a novel graph-based algorithm denoted as GLSM (Graph-based Line Segment Matching) is proposed in this paper, which includes: (1) the employment of three geometry types, i.e., homography, epipolar, and trifocal tensor, to constrain line and point candidates across views; (2) the method of unifying different geometry constraints into a line-point association graph for two or multiple views; and (3) a set of procedures for ranking, assigning, and clustering with the line-point association graph. The experimental results indicate that GLSM can obtain sufficient matches with a satisfactory accuracy in both two and multiple views. Moreover, GLSM can be employed with large image datasets. The implementation of GLSM will be available soon at https://skyearth.org/research/.

Robust line segment matching via reweighted random walks on the homography graph

This paper presents a novel method for matching line segments between stereo images. Given the fundamental matrix, the local homography can be over determined with pairwise line segment candidates. We exploit this constraint to initialize the candidate and construct the novel homography graph. Because the constraint between the node is based on the epipolar geometry, the homography graph is invariant to the local projective transformation. We employ the reweighted random walk on the graph to rank the candidate, then, we propose the constrained-greedy algorithm to obtain the reliable match. To the best of our knowledge, this is the first study to embed the epipolar geometry into the graph matching theory for the line segment matching. When evaluated on the 32 image patches, our method outperformed the state of the art methods, especially in the scenes of the wide baseline, steep viewpoint changes and dense line segments. The proposed algorithm is available at https://github.com/weidong-whu/line-match-RRW.