Local Homography Research Articles

Mask-Homo: Pseudo Plane Mask-Guided Unsupervised Multi-Homography Estimation

Homography estimation is a fundamental problem in computer vision. Previous works mainly focus on estimating either a single homography, or multiple homographies based on mesh grid division of the image. In practical scenarios, single homography is inadequate and often leads to a compromised result for multiple planes; while mesh grid multi-homography damages the plane distribution of the scene, and does not fully address the restriction to use homography. In this work, we propose a novel semantics guided multi-homography estimation framework, Mask-Homo, to provide an explicit solution to the multi-plane depth disparity problem. First, a pseudo plane mask generation module is designed to obtain multiple correlated regions that follow the plane distribution of the scene. Then, multiple local homography transformations, each of which aligns a correlated region precisely, are predicted and corresponding warped images are fused to obtain the final result. Furthermore, a new metric, Mask-PSNR, is proposed for more comprehensive evaluation of alignment. Extensive experiments are conducted to verify the effectiveness of the proposed method. Our code is available at https://github.com/SAITPublic/MaskHomo.

Robust Data Association Against Detection Deficiency for Semantic SLAM

Robust and accurate object association is essential for precise 3D object landmark inference in semantic Simultaneous Localization and Mapping (SLAM), and yet remains challenging due to the detection deficiency caused by high miss detection rate, false alarm, occlusion and limited field-of-view, etc. The 2D location of an object is a crucial complementary cue to the appearance feature, especially in the case of associating objects across frames under large viewpoint changes. However, motion model or trajectory pattern based methods struggle to infer object motion reliably with a moving camera. In this paper, by exploiting the local projective warping consistency, a local homography based 2D motion inference method is proposed to sequentially estimate the object location along with uncertainty. By integrating the deep appearance feature and semantic information, an object association method, named HOA, which is robust to detection deficiency is proposed. Experimental evaluations suggest that the proposed motion prediction method is capable of maintaining a low cumulative error over a long duration, which enhances the object association performance in both accuracy and robustness. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work aims to consistently associate 2D detection boxes corresponding to the same 3D object across images. In tasks of landmark-based navigation, collision avoidance, grasping and manipulation, objects in the task space are commonly simplified into 3D enveloping surfaces (e.g. cuboid or ellipsoid) by using 2D object detection boxes from multiple image views, and accurate data association is a prerequisite for precise enveloping surface reconstruction. This problem remains challenging considering the imperfect object detections, the appearance similarity of objects and the unpredictable trajectory of the moving camera. This work proposes a long-term reliable 2D location prediction algorithm that is capable of handling the complex motion of the target. Along with the appearance feature extracted by a retrain-free deep learning based model, this work proposes an object association method that can simultaneously deal with multiple objects with unknown object categories under the moving camera scenario.

Multi-angle Scheimpflug projection 3D microscope: Design, calibration, and three-dimensional reconstruction

3D measurement of complex micro-structures has becoming increasingly important for precision manufacturing. In this paper, a multi-angle Scheimpflug projection 3D microscope (MASP-3DM) is proposed, and the detailed design procedure is given. The system deployed four Scheimpflug projections and one telecentric imaging to overcome the shortcomings of limited depth of field (DOF), occlusion and reflection. Then, an improved and practical system calibration method is proposed, which utilizes pinhole projection model to calibrate the telecentric imaging. Meanwhile, local homography is combined with perspective-n-point (PnP) strategy to improve the calibration accuracy. Further, a point-cloud registration error correction model is established to achieve complete and high-precision measurement. The experimental results indicate that the re-projection error is within ±0.5μm and the registration error is less than 2μm for system calibration, the maximum of mean deviation, measurement bias, standard deviation, and type A uncertainty are respectively less than 3μm, 7μm, 4μm, and 1μm for 3D measurement, within the 10(H)mm×50(W)mm×40(D)mm volume.

Robust OCR Pipeline for Automated Digitization of Mother and Child Protection Cards in India

The Universal Immunization Programme in India has a mandate to fully vaccinate all of India’s 27 million children born annually. The vaccination doses are recorded by frontline health workers on standardized paper-based Mother and Child Protection (MCP) cards, which are manually digitized by data entry operators, resulting in poor data quality, delays, and significant time and resources. In our article, we focus on Optical Character Recognition– (OCR) based automated digitization of MCP card images captured through a smartphone application developed by us. By utilizing a standardized template for the MCP cards, which is available a priori , we register the card images and perform OCR on the extracted region of interest (ROIs). Since the cards with curvature or torn edges had poor ROIs, we built a global–local alignment technique that first approximates the ROI using global homography and then refines using a local homography resulting in improved accuracy. Our pipeline gives a character level accuracy of 98.73% on our dataset against 75.02% by Google Cloud Vision and 79.26% by Azure OCR. We also describe our field testing experience, where the digitized MCP card images were used to provide useful features on the smartphone application for health workers to conduct vaccination sessions.

Hierarchical Edge-Preserving Dense Matching by Exploiting Reliably Matched Line Segments

Image dense matching plays a crucial role in the reconstruction of three-dimensional models of buildings. However, large variations in target heights and serious occlusion lead to obvious mismatches in areas with discontinuous depths, such as building edges. To solve this problem, the present study mines the geometric and semantic information of line segments to produce a constraint for the dense matching process. First, a disparity consistency-based line segment matching method is proposed. This method correctly matches line segments on building structures in discontinuous areas based on the assumption that, within the corresponding local areas formed by two corresponding line pairs, the disparity obtained by the coarse-level matching of the hierarchical dense matching is similar to that derived from the local homography estimated from the corresponding line pairs. Second, an adaptive guide parameter is designed to constrain the cost propagation between pixels in the neighborhood of line segments. This improves the rationality of cost aggregation paths in discontinuous areas, thereby enhancing the matching accuracy near building edges. Experimental results using satellite and aerial images show that the proposed method efficiently obtains reliable line segment matches at building edges with a matching precision exceeding 97%. Under the constraint of the matched line segments, the proposed dense matching method generates building edges that are visually clearer, and achieves higher accuracy around edges, than without the line segment constraint.

Pixel-Wise Warping for Deep Image Stitching

Existing image stitching approaches based on global or local homography estimation are not free from the parallax problem and suffer from undesired artifacts. In this paper, instead of relying on the homography-based warp, we propose a novel deep image stitching framework exploiting the pixel-wise warp field to handle the large-parallax problem. The proposed deep image stitching framework consists of a Pixel-wise Warping Module (PWM) and a Stitched Image Generating Module (SIGMo). For PWM, we obtain pixel-wise warp in a similar manner as estimating an optical flow (OF). In the stitching scenario, the input images usually include non-overlap (NOV) regions of which warp cannot be directly estimated, unlike the overlap (OV) regions. To help the PWM predict a reasonable warp on the NOV region, we impose two geometrical constraints: an epipolar loss and a line-preservation loss. With the obtained warp field, we relocate the pixels of the target image using forward warping. Finally, the SIGMo is trained by the proposed multi-branch training framework to generate a stitched image from a reference image and a warped target image. For training and evaluating the proposed framework, we build and publish a novel dataset including image pairs with corresponding pixel-wise ground truth warp and stitched result images. We show that the results of the proposed framework are quantitatively and qualitatively superior to those of the conventional methods.

Unsupervised Global and Local Homography Estimation with Motion Basis Learning.

In this paper, we introduce a new framework for unsupervised deep homography estimation. Our contributions are 3 folds. First, unlike previous methods that regress 4 offsets for a homography, we propose a homography flow representation, which can be estimated by a weighted sum of 8 pre-defined homography flow bases. Second, considering a homography contains 8 Degree-of-Freedoms (DOFs) that is much less than the rank of the network features, we propose a Low Rank Representation (LRR) block that reduces the feature rank, so that features corresponding to the dominant motions are retained while others are rejected. Last, we propose a Feature Identity Loss (FIL) to enforce the learned image feature warp-equivariant, meaning that the result should be identical if the order of warp operation and feature extraction is swapped. With this constraint, the unsupervised optimization can be more effective and the learned features are more stable. With global-to-local homography flow refinement, we also naturally generalize the proposed method to local mesh-grid homography estimation, which can go beyond the constraint of a single homography. Extensive experiments are conducted to demonstrate the effectiveness of all the newly proposed components, and results show that our approach outperforms the state-of-the-art on the homography benchmark dataset both qualitatively and quantitatively. Code is available at https://github.com/megvii-research/BasesHomo.

An adaptive feature region-based line segment matching method for viewpoint-changed images with discontinuous parallax and poor textures

Many methods have been proposed to extract line segment (LS) correspondences for images with viewpoint variations. However, the matching performance on images with viewpoint variations is still limited due to the influence of discontinuous parallax and poor textures on images. Existing line segment matching (LSM) methods rarely consider the coexistence of these two situations. This study attempts to address this problem by proposing an adaptive feature region (FR)-based LSM method based on the observation that different FRs are suitable for different image scenes under image viewpoint change. In the proposed method, LSs are paired, and two types of FRs, symmetric FR centered on the line-pair intersection and asymmetric FR with the intersection as a vertex, are constructed for every line pair. The asymmetric FR of a line pair is a subregion of its symmetric FR. And the complement region of asymmetric FR in symmetric FR is called complement FR. To choose a suitable FR for the matching of each line pair adaptively, the asymmetric FR-based feature descriptor similarity and epipolar line-based constraints are combined to determine candidate line pair matches firstly. Then, the candidate matches with similar complement FR-based feature descriptors are considered initial matches and used to construct a topological constraint to check other candidates. Finally, we estimate an adaptive influence region-based local homography for each matched line pair to constrain the matching of unmatched individual LSs. The experimental results on the test images with viewpoint changes show that our approach outperforms the state-of-the-art methods in terms of Recall, matching precision (MP), and F-Measure (reflecting the overall matching performance of Recall and MP). Especially, it improves the average Recall and average F-Measure of the best method among the comparison methods by 45.20% and 25.83%, respectively.

Accurate and flexible calibration method for a 3D microscopic structured light system with telecentric imaging and Scheimpflug projection.

3D imaging and metrology of complex micro-structures is a critical task for precision manufacturing and inspection. In this paper, an accurate and flexible calibration method for 3D microscopic structured light system with telecentric imaging and Scheimpflug projector is proposed. Firstly, a fringe projection 3D microscopy (FP-3DM) system consisting of a telecentric camera and a Scheimpflug projector is developed, which can take full advantage of the depth of field (DOF) and increase the measurement depth range. Secondly, an accurate and flexible joint calibration method is proposed to calibrate the developed system, which utilizes the established pinhole imaging model and Scheimpflug distortion model to calibrate telecentric imaging, and fully considers the correction and error optimization of the Scheimpflug projection model. Meanwhile, the optimized local homography is calculated to obtain more accurate sub-pixel correspondence between the camera and the projector, and the perspective-n-point (PnP) method make the 3D coordinate estimation of the feature point more accurate. Finally, a prototype and a dedicated calibration program are developed to realize high-resolution and high-precision 3D imaging. The experimental results demonstrate that the re-projection error is less than 1µm, and the 3D repeated measurement error based on feature fitting is less than 4µm, within the calibrated volume of 10(H)mm × 50(W)mm × 40(D)mm.

Geological Borehole Video Image Stitching Method Based on Local Homography Matrix Offset Optimization.

Due to the influence of the shooting environment and inherent image characteristics, there is a large amount of interference in the process of image stitching a geological borehole video. To accurately match the acquired image sequences in the inner part of a borehole, this paper presents a new method of stitching an unfolded borehole image, which uses the image generated from the video to construct a large-scale panorama. Firstly, the speeded-up robust feathers (SURF) algorithm is used to extract the image feature points and complete the rough matching. Then, the M-estimator sample consensus (MSAC) algorithm is introduced to remove the mismatched point pairs and obtain the homography matrix. Subsequently, we propose a local homography matrix offset optimization (LHOO) algorithm to obtain the optimal offset. Finally, the above process is cycled frame by frame, and the image sequence is continuously stitched to complete the construction of a cylindrical borehole panorama. The experimental results show that compared with those of the SIFT, Harris, ORB and SURF algorithms, the matching accuracy of our algorithm has been greatly improved. The final test is carried out on 225 consecutive video frames, and the panorama has a good visual effect, and the average time of each frame is 100 ms, which basically meets the requirements of the project.

Local Homography Estimation on User-Specified Textureless Regions

Local Homography Estimation on User-Specified Textureless Regions

Method for online matching of multiple identical targets using local homography matrix based on moving DLT

In the field of geometric measurement of stereo vision, problems such as interference sources and similarity of target characteristics increase the difficulty of the matching of multiple identical targets. An online matching method for multiple identical targets using local homography matrix based on moving direct linear transform is presented in this paper. Firstly, the matching feature point pairs between left and right background images are obtained in advance. Secondly, the left background image is evenly divided into multiple rectangular meshes, and the local homography matrix of each rectangular mesh is calculated offline. Thirdly, the mapping point of each target is estimated online by the local homography matrix corresponding to the rectangular mesh where the target is located, and the target point with the closest Euclidean distance to the mapping point is considered as the matching point. Computer simulation and quality experimental data proves effectiveness of the proposed method. Experimental results reflect that our method is better than existing matching methods based on Delaunay triangulation, and the root mean square error of image coordinates between the mapping points from our method and the actual matching points is less than 13 pixels when entire image size is 1920 pixels × 1200 pixels.

Global Color Consistency Correction for Large-Scale Images in 3-D Reconstruction

Global color consistency correction for multiview images in three-dimensional (3-D) reconstruction is an important problem. The color differences between the images will affect the result of dense matching, thereby reducing the geometric accuracy of the mesh model. Moreover, it will also affect the result of texture mapping, causing color differences in the textured model. The color correction method based on global optimization is mainly used to solve this problem. And existing methods usually use sparse matching points as the color correspondences, but the correction results are not accurate enough as a result of the sparsity of the points. Besides, their efficiency of solving large-scale images globally is low. This article proposes a novel color correction method to eliminate the color differences between large-scale multiview images effectively. The core idea of our method is to group images by graph partition algorithm, and then perform intragroup correction and intergroup correction in sequence. First, for each pair of images, we calculate the reliable matching regions around the sparse points as the color correspondences according to the local homography principle. Compared with sparse matching points, our strategy can achieve more accurate color correction results. Next, for large-scale images, we partition them into many groups. For each group of images, the correction parameters are solved to eliminate the color differences of the images included in the group. Finally, we eliminate the color differences between groups by intergroup correction to achieve overall color consistency. Experimental results on typical datasets demonstrate that the proposed method is better than the current representative methods. The proposed method shows better color consistency in the extreme cases, and also exhibits higher computational efficiency on large-scale image sets.

Hierarchical line segment matching for wide-baseline images via exploiting viewpoint robust local structure and geometric constraints

Line segment matching for wide-baseline images is challenging due to the significant viewpoint differences. In this study, we propose a hierarchical line segment matching method based on viewpoint robust local structure and geometric constraints. In our approach, line segments are paired and classified into three types representing heuristically those with different level of matchability: structured line pairs (S-LPs), unstructured line pairs (U-LPs), and individual line segments (I-LSs). Accordingly, we design a hierarchical matching framework that consists of three matching layers respectively corresponding to the above three types: in the first layer, robust local structures are constructed for S-LPs. We match the S-LPs by measuring local structure similarity (LSS). In the second layer, we build a topological descriptor-based constraint based on the S-LP matches and combine it with epipolar geometry-based constraints to select candidate matches for U-LPs, and then use LSS to further refine the matches. In the third layer, we estimate local homography for I-LSs to build constraints, to extract matches by exploiting the implicit region information of line pair matches. A pair of pre- and postprocessing algorithms, namely line segment merging and match reassignment, are performed before and after the matching procedure to overcome the negative effect of line segment fragmentation on the matching. Experimental results demonstrate that the proposed method performs better than state-of-the-art methods (with the largest improvement of 124.17% in terms of F-Measure over the best one among the compared methods).

UAV Image Stitching Based on Mesh-Guided Deformation and Ground Constraint

This article introduces a drone image stitching based on mesh-guided deformation and ground constraint, which can closely match the characteristics of images and achieve precise registration and acquire ideal stitching effect. The traditional methods use the homography model to align the image, which causes artifacts in the result of stitching the images with parallax. To overcome this situation, the image is divided into meshes and the mesh vertices of the target image are used to guide the warping. A new energy function is designed to represent the deformation characteristics of the image. We propose a new alignment term by using local homography and a local scale term by using the edge information of the mesh. The established mesh-guided deformation model can overcome image parallax caused by some external factors and eliminate the ghostly parts of the result. Moreover, imaged scene is not effectively planar and some fluctuations exist in the scene of the images, which will distort the stitching result. We propose a ground constraint with the ground plane as the main plane to reduce projection distortions in non-overlapping areas between images. Finally, the method of creating groundtruth is proposed, which can evaluate the naturalness of results and make comparison more reasonable. Several sets of challenging drone images are tested, and the experimental results show that our stitching system has good results.

Infrared video image stabilization algorithm based on joint camera path

To solve the problem of infrared video image stabilization caused by parallax variation, the infrared video image stabilization algorithm based on joint camera path was proposed, which included four steps: preprocessing, joint camera path solving, multi-path optimization and motion compensation. Firstly, histogram equalization, feature point extraction, matching processing and pre-mapping were needed for infrared image. Then, each frame of image was divided into m×n grid cells, and local homography matrixs obtained from the corresponding grid in each frame were multiplied by the mapping motion representation based on the grid to obtain the joint camera path. Then, the path of the joint camera was smoothed by the strategy of ‘multi-path optimization’. Finally, the smooth path was used to stabilize the video. The experimental results show that this method can effectively deal with the nonlinear motion caused by parallax, which is better than the traditional image stabilization algorithm, and can achieve good image stabilization even when the feature points have partial occlusion.

Multiframe Astronomical Image Registration Based on Block Homography Estimation

Due to the influence of atmospheric turbulence, a time-variate video of an observed object by using the astronomical telescope drifts randomly with the passing of time. Thereafter, a series of images is obtained snapshotting from the video. In this paper, a method is proposed to improve the quality of astronomical images only through multiframe image registration and superimposition for the first time. In order to overcome the influence of anisoplanatism, a specific image registration algorithm based on multiple local homography transformations is proposed. Superimposing registered images can achieve an image with high definition. As a result, signal-to-noise ratio, contrast-to-noise ratio, and definition are improved significantly.

Robust line segment matching via reweighted random walks on the homography graph

This paper presents a novel method for matching line segments between stereo images. Given the fundamental matrix, the local homography can be over determined with pairwise line segment candidates. We exploit this constraint to initialize the candidate and construct the novel homography graph. Because the constraint between the node is based on the epipolar geometry, the homography graph is invariant to the local projective transformation. We employ the reweighted random walk on the graph to rank the candidate, then, we propose the constrained-greedy algorithm to obtain the reliable match. To the best of our knowledge, this is the first study to embed the epipolar geometry into the graph matching theory for the line segment matching. When evaluated on the 32 image patches, our method outperformed the state of the art methods, especially in the scenes of the wide baseline, steep viewpoint changes and dense line segments. The proposed algorithm is available at https://github.com/weidong-whu/line-match-RRW.

Image Registration Algorithm Based on Convolutional Neural Network and Local Homography Transformation

In order to overcome the poor robustness of traditional image registration algorithms in illuminating and solving the problem of low accuracy of a learning-based image homography matrix estimation algorithm, an image registration algorithm based on convolutional neural network (CNN) and local homography transformation is proposed. Firstly, to ensure the diversity of samples, a sample and label generation method based on moving direct linear transformation (MDLT) is designed. The generated samples and labels can effectively reflect the local characteristics of images and are suitable for training the CNN model with which multiple pairs of local matching points between two images to be registered can be calculated. Then, the local homography matrices between the two images are estimated by using the MDLT and finally the image registration can be realized. The experimental results show that the proposed image registration algorithm achieves higher accuracy than other commonly used algorithms such as the SIFT, ORB, ECC, and APAP algorithms, as well as another two learning-based algorithms, and it has good robustness for different types of illumination imaging.

UAV Image Mosaicking Based on Multiregion Guided Local Projection Deformation

The goal of unmanned aerial vehicle (UAV) image mosaicking is to create natural- looking mosaics without artifacts due to the parallax of the image and relative camera motion. UAV remote sensing is a low-altitude technology and the UAV imaged scene is not effectively planar, yielding parallax on the images. Moreover, when an object in 3-D is mapped to an image plane, different surfaces have different projections. These projections vary with the viewpoint in a sequence of UAV images, which causes artifacts near some tall buildings in the stitched images. To solve these problems, we propose a novel stitching method based on multiregion guided local projection deformation, which can significantly reduce ghosting due to these projections vary with the viewpoint and the parallax. In the proposed method, the image is initially meshed and each cell corresponds to a local homography for image matching, which can reduce misalignment artifacts in the results compared with 2-D projective transforms or global homography. Then, we divide the overlapping regions of input images into multiple regions by classifying feature points. The partitioned regions which serve well scene constraints, are employed to guide the calculation of local homography. Specifically, instead of calculating local homography by the distance between all the feature points in the image and the vertices of the grid, we propose a strategy where multiple regions have different weights for calculating local homography, which can significantly reduce ghosting near some tall buildings. The benefits of the proposed approach are demonstrated using a variety of challenging cases.