Homography Transformation Research Articles

Research on intelligent auxiliary assembly technology based on deep learning

Background Auxiliary assembly refers to guiding and prompting the assembly process to help operators complete complex assembly operations. Due to the complex structure of products, the similar shape of parts and human factors, the misassembly and missing assembly of parts still occur in the process of product assembly, so it is of great significance to detect the assembly correctness of complex products. Methods Aiming at the problem that manual inspection is inefficient and depends heavily on the level of inspectors in the process of complex product assembly inspection, this paper proposes an assembly correctness detection method based on deep learning. Through the three steps of view transformation, semantic segmentation and template matching, the automatic judgment of assembly errors such as wrong assembly, missing assembly and redundancy is realized, and the method is verified by the computer motherboard. Results Taking the computer motherboard as the verification object to test the correctness of assembly, the experimental re sults show that the perspective adjustment of the image after homography transformation is very obvious. The evaluation index of the semantic segmentation network detection object is calculated, and each accuracy meets the requirements of assembly correctness detection. A visualization module is also used to visually display the results of assembly correctness detection based on template matching. Conclusions The assembly correctness detection method can provide a guarantee for the manual assembly process and reduce the error rate of assembly. The machine vision detection technology can be used for automatic detection of assembly quality to improve the efficiency and automation level of detection.

HyperMRI: hyperspectral and magnetic resonance fusion methodology for neurosurgery applications

PurposeMagnetic resonance imaging (MRI) is a common technique in image-guided neurosurgery (IGN). Recent research explores the integration of methods like ultrasound and tomography, among others, with hyperspectral (HS) imaging gaining attention due to its non-invasive real-time tissue classification capabilities. The main challenge is the registration process, often requiring manual intervention. This work introduces an automatic, markerless method for aligning HS images with MRI.MethodsThis work presents a multimodal system that combines RGB-Depth (RGBD) and HS cameras. The RGBD camera captures the patient’s facial geometry, which is used for registration with the preoperative MR through ICP. Once MR-depth registration is complete, the integration of HS data is achieved using a calibrated homography transformation. The incorporation of external tracking with a novel calibration method allows camera mobility from the registration position to the craniotomy area. This methodology streamlines the fusion of RGBD, HS and MR images within the craniotomy area.ResultsUsing the described system and an anthropomorphic phantom head, the system has been characterised by registering the patient’s face in 25 positions and 5 positions resulted in a fiducial registration error of 1.88 ± 0.19 mm and a target registration error of 4.07 ± 1.28 mm, respectively.ConclusionsThis work proposes a new methodology to automatically register MR and HS information with a sufficient accuracy. It can support the neurosurgeons to guide the diagnosis using multimodal data over an augmented reality representation. However, in its preliminary prototype stage, this system exhibits significant promise, driven by its cost-effectiveness and user-friendly design.

Development of Multi-Strip Image Mosaicking for KOMPSAT-3A Images

Abstract. High-resolution satellite imagery has a limitation in terms of coverage area. This limitation presents challenges for extensive-scale analysis at regional or national levels. To maximize the utility of high-resolution satellite imagery, the implementation of image mosaicking techniques is essential. In this paper, we have developed seamline extraction techniques and relative geometric correction optimized for high-resolution satellite imagery. Ultimately, we proposed a multi-strip image mosaicking method for KOMPSAT-3A (Korea Multi-Purpose Satellite-3A) images. We applied the Dijkstra's shortest path algorithm to efficiently extract seamlines. we also performed image registration based on feature matching and homography transformation to correct the relative geometric errors between input images. We conducted experiments with our methods using 29 scenes from KOMPSAT-3A L1G data. The results indicated high relative geometric accuracy, with an average error of 1.63 pixels. Furthermore, we were able to obtain high-quality seamless mosaic images. Our proposed method is expected to enhance the utility of KOMPSAT-3A imagery for large-scale environmental and urban analysis and to provide more accurate and comprehensive data.

Mask-Homo: Pseudo Plane Mask-Guided Unsupervised Multi-Homography Estimation

Homography estimation is a fundamental problem in computer vision. Previous works mainly focus on estimating either a single homography, or multiple homographies based on mesh grid division of the image. In practical scenarios, single homography is inadequate and often leads to a compromised result for multiple planes; while mesh grid multi-homography damages the plane distribution of the scene, and does not fully address the restriction to use homography. In this work, we propose a novel semantics guided multi-homography estimation framework, Mask-Homo, to provide an explicit solution to the multi-plane depth disparity problem. First, a pseudo plane mask generation module is designed to obtain multiple correlated regions that follow the plane distribution of the scene. Then, multiple local homography transformations, each of which aligns a correlated region precisely, are predicted and corresponding warped images are fused to obtain the final result. Furthermore, a new metric, Mask-PSNR, is proposed for more comprehensive evaluation of alignment. Extensive experiments are conducted to verify the effectiveness of the proposed method. Our code is available at https://github.com/SAITPublic/MaskHomo.

Rapid full-field deformation measurements of tall buildings using UAV videos and deep learning

Full-field deformation of tall buildings is informative and imperative for structural health condition assessments. This study explores the feasibility of using videos recorded by an unmanned aerial vehicle (UAV) to measure the vibration profiles of tall buildings with the aid of deep neural networks. First, a Transformer network with a self-attention mechanism is used, and its outcome is employed for the homography transformation to minimize the drifting problems induced by the UAV. Second, a boundary-aware salient object detection network is adopted to extract the boundary of the buildings in the stabilized videos for further full-field deformation measurements. The accuracy of the proposed method is experimentally validated using a wood frame structure under harmonic excitations. Under a measuring distance of approximately 4.2 m, the proposed method achieves averages of root-mean-square-errors (RMSEs) for the displacement and rotation measurements of 2.13 mm and 1.74 × 10−3 rad, respectively, as compared to that determined by an infrared light tracking system. The study further analyzes the field-test data recorded from the vibration of a 247-m tower. Under a measuring distance of about 55 m, the proposed method is comparable to the vibration measurement at a specific story and estimates the full-field deformation within the region of interest. Overall, the proposed method with rapid UAV deployment is feasible and accurate for monitoring the full-field deformation using the UAV-recorded videos.

Research on intelligent auxiliary assembly technology based on deep learning

Background Auxiliary assembly refers to guiding and prompting the assembly process to help operators complete complex assembly operations. Due to the complex structure of products, the similar shape of parts and human factors, the misassembly and missing assembly of parts still occur in the process of product assembly, so it is of great significance to detect the assembly correctness of complex products. Methods Aiming at the problem that manual inspection is inefficient and depends heavily on the level of inspectors in the process of complex product assembly inspection, this paper proposes an assembly correctness detection method based on deep learning. Through the three steps of view transformation, semantic segmentation and template matching, the automatic judgment of assembly errors such as wrong assembly, missing assembly and redundancy is realized, and the method is verified by the computer motherboard. Results Taking the computer motherboard as the verification object to test the correctness of assembly, the experimental re sults show that the perspective adjustment of the image after homography transformation is very obvious. The evaluation index of the semantic segmentation network detection object is calculated, and each accuracy meets the requirements of assembly correctness detection. A visualization module is also used to visually display the results of assembly correctness detection based on template matching. Conclusions The assembly correctness detection method can provide a guarantee for the manual assembly process and reduce the error rate of assembly. The machine vision detection technology can be used for automatic detection of assembly quality to improve the efficiency and automation level of detection.

Three-dimension deformation measurement using modified microscopic stereo-vision technology

An improved microscopic stereo-vision technology was developed for microscopic deformation (displacement and strain) measurement. First, the distortion of the microscope lens is significant, but the existing internal parameters calibration approach is prone to abnormal calibration results due to the parameter coupling. An innovative calibration method based on a modified homography transformation model is proposed. Second, the short working distance and the stereo angle of the two cameras lead to the degradation of image quality, and the correlation analysis is challenged. The genetic algorithm (GA) was modified and introduced in three-dimensional digital image correlation (3D-DIC) to help search for more accurate initial values of the displacement. The proposed GA-assisted DIC shows higher subpixel precision and speed in the image-matching test. Then, the developed microscopic stereo-vision technology incorporating the above innovative innovations is applied to a mechanical properties experiment with a 3D-printed artificial bone scaffold.

Geolocalization from Aerial Sensing Images Using Road Network Alignment

Estimating the geographic positions in GPS-denied environments is of great significance to the safe flight of unmanned aerial vehicles (UAVs). In this paper, we propose a novel geographic position estimation method for UAVs after road network alignment. We discuss the generally overlooked issue, namely, how to estimate the geographic position of the UAV after successful road network alignment, and propose a precise robust solution. In our method, the optimal initial solution of the geographic position of the UAV is first estimated from the road network alignment result, which is typically presented as a homography transformation between the observed road map and the reference one. The geographic position estimation is then modeled as an optimization problem to align the observed road with the reference one to improve the estimation accuracy further. Experiments on synthetic and real flight aerial image datasets show that the proposed algorithm can estimate more accurate geographic position of the UAV in real time and is robust to the errors from homography transformation estimation compared to the currently commonly-used method.

RELATIVE GEOMETRIC CORRECTION OF MULTIPLE SATELLITE IMAGES BY RIGOROUS BLOCK ADJUSTMENT

Abstract. With the development of satellite imaging technology, many Earth observation satellites have been launched and almost all areas of the Earth are daily covered by satellite images. It is promising to utilize such images in change detection, monitoring, semantic classification, etc. To obtain geometric information on each pixel in images, Rational Polynomial Coefficients (RPCs) of a Rational Functional Model (RFM) are provided with satellite images. However initial RPCs may include geometric errors. Therefore, errors in initial RPCs must be corrected before utilization. In this study, we attempt to perform rigorous block adjustments for refining RFMs of overlapping satellite images without ground control points. Our rigorous block adjustment method operated based on automatically extracted tie-points within overlapping areas. It estimates the optimal adjustment coefficients of RPCs and ground coordinates of tie-points. We achieved relative geometric correction of multiple satellite images by transforming the images based on the estimated adjustment coefficients. As a result, an initial RFM model with an error of around 21.73 pixels was corrected to within 1 pixel, and the reprojection error of check points decreased to 0.87 pixels. We also confirmed that our method showed more accurate results than general image registration methods, such as 2D homography transformation.

Thermal-textured BIM generation for building energy audit with UAV image fusion and histogram-based enhancement

Mapping Unmanned Aerial Vehicle (UAV)-based thermal images of building façades onto a Building Information Model (BIM) can greatly support building energy audit. However, accurately registering these images into BIM is challenging due to thermal image’s unique characteristics of low texture and high distortion. To address these issues, this paper proposes a new multi-source image fusion framework for registering UAV thermal images to BIM. First, a low-cost target is designed and produced using 3D printing for calibrating UAV thermal camera. The rectified thermal images are then fused with RGB images through Homography transformation, enabling thermal images to be registered accurately with BIM. Additionally, a histogram-based approach is proposed to correct the thermally inconsistent phenomenon and enhance the contrast to produce high-contrast thermal-textured BIM. Field tests confirmed a mean image fusion error of around 5.713 pixels. A case study on a masonry building demonstrated the feasibility and effectiveness of the proposed framework.

Improved On-Orbit MTF Measurement Method Based on Point Source Arrays

The modulation transfer function (MTF) is a key characteristic used to assess the performance of optical remote sensing satellite sensors. MTF detection can directly measure a sensor’s two-dimensional (2D) point spread function (PSF); therefore, it has been applied to various high-resolution remote sensing satellites (e.g., Pleiades) using point sources. However, current point source methods mainly use 2D Gaussian functions to fit the discrete digital number (DN) of the point source on the image to extract the center of the point source and fit the PSF after encrypting multiple point sources; thus, noise robustness is poor and measurement accuracy varies widely. In this study, we developed a noise-resistant on-orbit MTF detection method based on the object space constraint among point source arrays. Utilizing object space constraint relationships among points in a point source array, a homography transformation model was established, enabling accurate extraction of sub-pixel coordinates for each point source response. Subsequently, aligning the luminosity distribution of all point sources concerning a reference point source, the encrypted PSF was obtained and then fitted to obtain the MTF. To validate the method, Gaofen-2 (GF-2) satellite images were used to conduct an in-orbit imaging experiment on the point source array of the Chinese Zhongwei remote sensing satellite calibration site. Compared with the Gaussian model methods, the proposed method yielded more accurate peak positions for each point source. Standard deviations of peak position constant ratios in along- and cross-track directions improved by 2.8 and 4.8 times, respectively. The root-mean-square error (RMSE) of the collinearity test results increased by 92%, and the noise resistance of the MTF curve improved by two times. Dynamic MTF values at the Nyquist frequency for the GF-2 panchromatic band in along- and cross-track directions were 0.0476 and 0.0705, respectively, and MTF values in different directions were well distinguished.

NTPP-MVSNet: Multi-View Stereo Network Based on Neighboring Tangent Plane Propagation

Although learning-based multi-view stereo algorithms have produced exciting results in recent years, few researchers have explored the specific role of deep sampling in the network. We posit that depth sampling accuracy more directly impacts the quality of scene reconstruction. To address this issue, we proposed NTPP-MVSNet, which utilizes normal vector and depth information from neighboring pixels to propagate tangent planes. Based on this, we obtained a more accurate depth estimate through homography transformation. We used deformable convolution to acquire continuous pixel positions on the surface and 3D-UNet to account for the regression of depth and normal vector maps without consuming additional GPU memory. Finally, we applied homography transformation to complete the mapping of the imaging plane and the neighborhood surface tangent plane to generate a depth hypothesis. Experimental trials on the DTU and Tanks and Temples datasets demonstrate the feasibility of NTPP-MVSNet, and ablation experiments confirm the superior performance of our deep sampling methodology.

Food Classification and Meal Intake Amount Estimation through Deep Learning

This paper proposes a method to classify food types and to estimate meal intake amounts in pre- and post-meal images through a deep learning object detection network. The food types and the food regions are detected through Mask R-CNN. In order to make both pre- and post-meal images to a same capturing environment, the post-meal image is corrected through a homography transformation based on the meal plate regions in both images. The 3D shape of the food is determined as one of a spherical cap, a cone, and a cuboid depending on the food type. The meal intake amount is estimated as food volume differences between the pre-meal and post-meal images. As results of the simulation, the food classification accuracy and the food region detection accuracy are up to 97.57% and 93.6%, respectively.

GEMVS: a novel approach for automatic 3D reconstruction from uncalibrated multi-view Google Earth images using multi-view stereo and projective to metric 3D homography transformation

ABSTRACT This paper proposes a novel approach for automatic 3D surface reconstruction from uncalibrated and multi-view Google Earth images by using a multi-view stereo method and 3D projective to metric transformation. Without the Rational Polynomial Coefficients, it is impossible to obtain the metric reconstruction of the 3D surface from multi-view satellite images. We solve the uncalibrated multi-view satellite image problem by employing a multi-view stereo vision technique followed by a projective to metric transformation. The virtual pose parameters of the satellite images are obtained by using COLMAP, and the virtual 3D projective reconstruction is done by using EnSoft3D. For projective to metric transformation, we propose to employ 3D homography transformation. Eight 3D correspondence pairs on the viewing frustums between the virtual reference camera and the ideal nadir camera are used to derive a 3D homography matrix. Using the 3D homography matrix, we finally obtain the metric reconstruction of 3D surface up to an unknown height scale in a reference coordinate system of the Google Earth desktop software. Experiments are done in several world locations on Google Earth including building and vegetation areas. Reconstruction error analysis with the Data Fusion Contest 19 dataset is also presented. The average of MAE and RMSE of five tile regions in the dataset are 1.596 m and 2.083 m, respectively.

Active Pose Relocalization for Intelligent Substation Inspection Robot

Intelligent inspection robots widely applied in substations are required to capture the inspection image consistent with the calibration image during routine inspection of electrical equipment. However, it is a challenging work for the inspection robot to capture the inspection image meeting the requirement due to navigation error and mechanical wear. To address this problem, an active pose relocalization (APR) method is proposed in this article. Specifically, an error model describing the relationship between the pixel error in the image plane and the robot pose error is established. Then, a decoupling three-stage PI control strategy based on the error model is provided to relocate the robot to the calibration pose, wherein, a translation error estimation algorithm based on homography transformation is proposed to compute the absolute translation scale between calibration and inspection poses, which avoid the degradation problem of the classic 2D-2D pose estimation algorithm. Finally, the performance of the proposed APR method is demonstrated through comparative relocalization experiments of 10 calibration points in virtual and real-world environments respectively.

A.I. Pipeline for Accurate Retinal Layer Segmentation Using OCT 3D Images

An image data set from a multi-spectral animal imaging system was used to address two issues: (a) registering the oscillation in optical coherence tomography (OCT) images due to mouse eye movement and (b) suppressing the shadow region under the thick vessels/structures. Several classical and A.I.-based algorithms, separately and in combination, were tested for each task to determine their compatibility with data from the combined animal imaging system. The hybridization of A.I. with optical flow followed by homography transformation was shown to be effective (correlation value > 0.7) for registration. Resnet50 backbone was shown to be more effective than the famous U-net model for shadow region detection with a loss value of 0.9. A simple-to-implement analytical equation was shown to be effective for brightness manipulation with a 1% increment in mean pixel values and a 77% decrease in the number of zeros. The proposed equation allows the formulation of a constraint optimization problem using a controlling factor α for the minimization of the number of zeros, the standard deviation of the pixel values, and maximizing the mean pixel value. For layer segmentation, the standard U-net model was used. The A.I.-Pipeline consists of CNN, optical flow, RCNN, a pixel manipulation model, and U-net models in sequence. The thickness estimation process had a 6% error compared with manually annotated standard data.

Homography Estimation Using Marker Projection Control: A Case of Calibration-Free Projection Mapping

Homography transformation, or planar projective transformation, between camera and projector images, which can be estimated by observing four or more pairs of corresponding points between two images, plays a crucial role in dynamic projection mapping with an uncalibrated projector-camera pair. This study discusses a case of dynamic projection mapping onto a moving textured surface where the point correspondence is provided through an actively controlled (not static) marker pattern projection by the projector along with the movement of the surface. Qualitative advantages are discussed in terms of system implementation, the image processing algorithm, and artifact perception by human observers. In addition, a quantitative analysis of alignment errors in the projection mapping is presented to show the effectiveness of the active control of marker patterns.

Image Stitching With Manifold Optimization

Image stitching usually relies on spatial transformations to do the overlap alignment and distortion mitigation. This paper presents a novel manifold optimization method to seek for the optimal transformations. Especially, the transformations involved in the stitching are treated as elements of a prescribed matrix manifold. Then, the spatially varying homographies are computed by efficient second-order minimization (ESM) on the geometric error of aligning feature correspondences, but with their intrinsic manifold parameterization. To mitigate the distortion, the interpolation between homography and similarity transformation is performed on a general matrix manifold. These on-manifold operations improve the stitching quality with less artifacts of ghosting and distortion. The experiments show the effectiveness of our manifold optimization for image stitching.

An Improved Speed Estimation Using Deep Homography Transformation Regression Network on Monocular Videos

Vehicle speed estimation is one of the most critical issues in intelligent transportation system (ITS) research, while defining distance and identifying direction have become an inseparable part of vehicle speed estimation. Despite the success of traditional and deep learning approaches in estimating vehicle speed, the high cost of deploying hardware devices to get all related sensor data, such as infrared/ultrasonic devices, Global Positioning Systems (GPS), Light Detection and Ranging (LiDAR systems), and magnetic devices, has become the key barrier to improvement in previous studies. In this paper, our proposed model consists of two main components: 1) a vehicle detection and tracking component – this module is designed for creating reliable detection and tracking every specific object without doing calibration; 2) homography transformation regression network – this module has a function to solve occlusion issues and estimate vehicle speed accurately and efficiently. Experimental results on two datasets show that the proposed method outperforms the state-of-the-art methods by reducing the mean square error (MSE) metric from 14.02 to 6.56 based on deep learning approaches. We have announced our test code and model on GitHub with <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ervinyo/Speed-Estimation-Using-Homography-Transformation-and-Regression-Network</uri> .

Automatic detection, classification and localization of defects in large photovoltaic plants using unmanned aerial vehicles (UAV) based infrared (IR) and RGB imaging

This study aims to build a photovoltaic (PV) plant maintenance and operation system, using an unmanned aerial vehicle (UAV) carrying a thermal imager to take images. In the proposed system, the infrared (IR) image was used for detecting PV module thermal defects, and the RGB image was used for detecting module surface defects. The two images were employed to cross validate the causes for module defects.In Part I, the PV plant information pattern was created, and the Taiwan PV plant (1,482 PV modules, 410 kW) was taken as an example. The PV system image feature points were detected by using the Scale Invariant Feature Transform (SIFT), in order to solve the feature variation problems, such as image luminance, rotation, and zoom in/out. The same feature points of multiple local power plant images were matched. Afterwards, the optimal number of feature points was calculated by homography transformation and random sample consensus (RANSAC) to form the PV plant panorama by image stitching. The PV plant panorama background noise was removed by image hue. The module segmentation of PV systems was performed by using image luminance, and the PV module was geometrically reconstructed by using morphology. The PV module edge contour was extracted by the Laplace operator to obtain the perimeter, area, and centroid features. The quantity and positions of PV modules were recognized and calculated to form the PV plant information pattern. In Part II, the PV module defect recognition and classification system was built. The defects in seven PV plants in Taiwan were collected, and the image features were enhanced using a convolutional neural network (CNN). Besides using the convolution layer to capture the image features, Max Pooling and local response normalization were used to enhance the image features. Color space transform was used to intensify the color features, increase the accuracy of the classification modules, and recognize and position the PV module defects. The IR image hot spot recognition accuracy was 100%. The classification accuracy of eight modules, including one normal module and seven defect modules, is 97.52%. The classification accuracy of six modules, including the appearances of one normal module and five defects in RGB images, is 99.17%. The classification accuracy of 14 defects in IR thermal images and RGB images is 97.52%. The causes of defects were cross validated by IR thermal image and RGB image. This study applied the K-fold cross validation to select the optimal model, and the recognition time of one image was shorter than 0.3 s, which is lower than the camera time constant. The results show that the system is applicable to real-time detections. In Part III, the PV plant defect information pattern was created. The PV module with defects was labeled during detection, and the defects in the power plant PV module and the positions thereof were obtained, which would be favorable for PV plant maintenance.