Image Alignment Method Research Articles

Accurate image alignment based on multi-warp optimization for large parallax

Image alignment with parallax is a challenging computer vision problem. While existing methods employing local-varying smooth warps have enhanced alignment accuracy for complex spacial deformations compared to global homography estimation, they fall short in adequately representing the discontinuous transformations evident in images with large parallax. In this paper, we propose an image alignment method grounded in the estimation and fusion of multiple warping models. To effectively tackle the challenges posed by discontinuous deformations in different regions, our method comprises two key stages: principal region alignment and fine region refinement. In both stages, multiple warping models are initially estimated using feature correspondences and are then concurrently optimized by minimizing pixel-level photometric loss. For each pixel, we select the optimal model that minimizes warping error. Additionally, we introduce a feature grouping method based on Delaunay triangulation. Experiments on real-world images demonstrate the superior alignment accuracy achieved by our proposed method compared to other state-of-the-arts.

Internet street view image fusion method using convolutional neural network

The use of image fusion technology in the area of information processing is continuing to advance in depth thanks to ongoing hardware advancements and related research. An enhanced convolutional neural network approach is developed to fuse visible and infrared images, and image pre-processing is carried out utilising an image alignment method with edge detection in order to gain more accurate and trustworthy image information. The performance of the fast wavelet decomposition, convolutional neural network, and modified convolutional neural network techniques is compared and examined using four objective assessment criteria. The experimental findings demonstrated that the picture alignment was successful with an offset error of fewer than 3 pixels in the horizontal direction and an angle error of less than 0.3∘ in both directions. The revised convolutional neural network method increased the information entropy, mean gradient, standard deviation, and edge detection information by an average of 46.13%, 39.40%, 19.91%, and 3.72%. The runtime of the modified approach was lowered by 19.42% when compared to the convolutional neural network method, which enhanced the algorithm’s performance and boosted the effectiveness of picture fusion. The image fusion accuracy reached 98.61%, indicating that the method has better fusion performance and is of practical value for improving image fusion quality.

Enhancing Image Alignment in Time-Lapse-Ground-Penetrating Radar through Dynamic Time Warping

Ground-penetrating radar (GPR) is a rapid and non-destructive geophysical technique widely employed to detect and quantify subsurface structures and characteristics. Its capability for time lapse (TL) detection provides essential insights into subsurface hydrological dynamics, including lateral flow and soil water distribution. However, during TL-GPR surveys, field conditions often create discrepancies in surface geometry, which introduces mismatches across sequential TL-GPR images. These discrepancies may generate spurious signal variations that impede the accurate interpretation of TL-GPR data when assessing subsurface hydrological processes. In responding to this issue, this study introduces a TL-GPR image alignment method by employing the dynamic time warping (DTW) algorithm. The purpose of the proposed method, namely TLIAM–DTW, is to correct for geometric mismatch in TL-GPR images collected from the identical survey line in the field. We validated the efficacy of the TLIAM–DTW method using both synthetic data from gprMax V3.0 simulations and actual field data collected from a hilly, forested area post-infiltration experiment. Analyses of the aligned TL-GPR images revealed that the TLIAM–DTW method effectively eliminates the influence of geometric mismatch while preserving the integrity of signal variations due to actual subsurface hydrological processes. Quantitative assessments of the proposed methods, measured by mean absolute error (MAE) and root mean square error (RMSE), showed significant improvements. After performing the TLIAM–DTW method, the MAE and RMSE between processed TL-GPR images and background images were reduced by 96% and 78%, respectively, in simple simulation scenarios; in more complex simulations, MAE declined by 27–31% and RMSE by 17–43%. Field data yielded reductions in MAE and RMSE of >82% and 69%, respectively. With these substantial improvements, the processed TL-GPR images successfully depict the spatial and temporal transitions associated with subsurface lateral flows, thereby enhancing the accuracy of monitoring subsurface hydrological processes under field conditions.

Three-dimensional model of normal human dermal tissue using serial tissue sections.

Background: This study aims to construct a three-dimensional model of skin dermis utilizing continuous tissue sections, with the primary objective of obtaining anatomical structure data for normal human dermal tissues. Methods: Normal skin tissue specimens were acquired, paraffin-embedded, and subjected to HE staining. Panoramic images of skin sections were captured using a microscope. Tissue section images were aligned using the SIFT and StackReg image alignment methods, with analysis conducted using the OpenCV module. Mimics17 software facilitated the reconstruction of the skin dermal 3D model, enabling the calculation of dermal porosity and the void diameter. Results: Panoramic skin slices exhibited high-resolution differentiation of dermal fibers and cellular structures. Both SIFT and StackReg image alignment methods yielded similar results, although the SIFT method demonstrated greater robustness. Successful reconstruction of the three-dimensional dermal structure was achieved. Quantitative analysis revealed a dermal porosity of 18.96 ± 4.41% and an average pore diameter of 219.29 ± 34.27 μm. Interestingly, the porosity of the dermis exhibited a gradual increase from the papillary layer to the fourth layer, followed by a transient decrease and then a gradual increase. The distribution of the mean pore diameter mirrored the pattern observed in porosity distribution. Conclusion: Utilizing the continuous skin tissue slice reconstruction technique, this study successfully reconstructed a high-precision three-dimensional tissue structure of the skin. The quantitative analysis of dermal tissue porosity and average pore diameter provides a standardized dataset for the development of biomimetic tissue-engineered skin.

Optimizing digital implant impressions: Evaluating the significance of scan body image deficiency and alignment under varied scan body exposures.

In implant dentistry, the advent of intraoral scanning technology has revolutionized traditional clinical processes by streamlining procedures and ensuring predictable treatment outcomes. However, achieving accurate virtual implant positions using intraoral scanners and scan bodies can be influenced by various clinical and laboratory factors. This study aims to investigate the impact of scan body image capture deficiency and scan body alignment methods in computer-aided design (CAD) software on the accuracy of virtual implant positions, particularly in different implant depths. Three stereolithographic half-arch implant models with different implant depths were prepared, representing three scenarios of scan body exposure: full exposed scan body, 2/3 exposed scan body, and 1/3 exposed scan body. The scan body image capture deficiency and alignment methods were simulated using CAD software. The deviation of virtual implant positions obtained from different scenarios were evaluated using 3D analysis software. The highest angular and linear deviation (0.237±0.059 degrees, 0.084±0.068 mm) were found in the 1/4 upper and lower part scan body deficiency using the 1-point alignment method in the 1/3 exposed scan body. Two-way ANOVA analysis revealed significant effects of scan deficiency on virtual implant position deviations across all scan body exposures, except for the linear deviation when the scan body was exposed 2/3 of its length. Furthermore, scan deficiencies in the 1/4 upper and lower parts of the scan body significantly affected implant angular deviation regardless of scan body exposure, while implant linear deviation was specifically affected when the scan body was exposed to only 1/3 of its total length. Deficiencies in scan body acquisition, particularly in deep soft tissue situations, can lead to deviations in both angular and linear positioning of virtual implants. Employing appropriate scan body alignment methods such as a 3-point alignment approach demonstrates better accuracy compared to a 1-point alignment.

Palm Print Database Collection System

Contactless biometric palm print recognition technology has attracted increased attention due to the covid-19 pandemic. Many dual camera based sensors have been proposed to capture palm vein and palm print image synchronously. However, translations between captured palm print images differ depending on the distance between the hand and the sensor. A biometric palm image Alignment method is proposed based on the imaging and ranging model. A palm print refers to an image acquired of the palm region of the hand. Palm print image enhancement is required for better feature extraction. To extract fine features enhancement is required. Fine features increase the accuracy of palm print recognition system. The main purpose of this review paper is to study and compare various palm print recognition techniques in terms of performance, accuracy, overheads and Peak Signal to Noise Ratio (PSNR)

A Backprojection-Based Autofocus Imaging Method for Circular Synthetic Aperture Radar

Circular synthetic aperture radar (CSAR) can obtain higher image resolution and more target information through 360° observation of the target. However, CSAR is often subject to motion errors due to the sensor’s accuracy limitations and the unique trajectory of the system. The autofocusing back projection algorithm based on the optimization of image sharpness compensates for motion errors through phase-error estimation. This method can attain relatively good performance while assuming the same error for all pixels; it ignores the spatial variance of motion errors. In order to solve this problem, this paper proposes an autofocusing method based on the Prewitt operator and particle swarm optimization (PSO), which first extracts the subaperture image feature points as a new dataset using the Prewitt operator, estimates the radar slope error on the new dataset using PSO, and subsequently compensates the subaperture image. Finally, the subaperture images are synthesized using an image-alignment method. The results of both the simulated experiments and the measured data validate the effectiveness of this method.

Cross‐class pest and disease vegetation detection based on small sample registration

AbstractThis paper introduces few‐shot anomaly detection (FSAD), a practical and less anomaly detection (AD) method, which can provide a limited number of normal images for each class during training. So far, studies on FSAD have been carried out according to each model, and there is no discussion of commonalities between different types. Depending on how people detect unusual lies, the problematic images are compared to the normal ones. The image alignment method based on different classifications is used to train the target detection model independent of classification, and performed ablation experiments on the pest and disease datasets in different environments for verification. This is the first time the FSAD method has been used to train a single scalable model without the need to train new classifications or adjust parameters. The experimental results show that the application of AUC based on vegetation disease data set and vegetation pest data set in FSAD algorithm is improved by 19.5% compared with the existing algorithm.

CyTran: A cycle-consistent transformer with multi-level consistency for non-contrast to contrast CT translation

We propose a novel approach to translate unpaired contrast computed tomography (CT) scans to non-contrast CT scans and the other way around. Solving this task has two important applications: (i) to automatically generate contrast CT scans for patients for whom injecting contrast substance is not an option, and (ii) to enhance the alignment between contrast and non-contrast CT by reducing the differences induced by the contrast substance before registration.Our approach is based on cycle-consistent generative adversarial convolutional transformers, for short, CyTran. Our neural model can be trained on unpaired images, due to the integration of a multi-level cycle-consistency loss. Aside from the standard cycle-consistency loss applied at the image level, we propose to apply additional cycle-consistency losses between intermediate feature representations, which enforces the model to be cycle-consistent at multiple representations levels, leading to superior results. To deal with high-resolution images, we design a hybrid architecture based on convolutional and multi-head attention layers. In addition, we introduce a novel data set, Coltea-Lung-CT-100W, containing 100 3D triphasic lung CT scans (with a total of 37,290 images) collected from 100 female patients (there is one examination per patient). Each scan contains three phases (non-contrast, early portal venous, and late arterial), allowing us to perform experiments to compare our novel approach with state-of-the-art methods for image style transfer.Our empirical results show that CyTran outperforms all competing methods. Moreover, we show that CyTran can be employed as a preliminary step to improve a state-of-the-art medical image alignment method. We release our novel model and data set as open source at: https://github.com/ristea/cycle-transformer.Our qualitative and subjective human evaluations reveal that CyTran is the only approach that does not introduce visual artifacts during the translation process. We believe this is a key advantage in our application domain, where medical images need to precisely represent the scanned body parts.

S$^{2}$ Loop: A Lightweight Spectral-Spatio Loop Closure Detector for Resource-Constrained Platforms

Visual loop closure detection is an essential backend task for long-term vSLAM applications. However, prior works cannot simultaneously meet the requirements of high recall and low computing and memory overhead, which prohibits their applicability to resource-constrained platforms. In this work we propose S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop, a novel spectral-spatio loop detector that explores an efficient loop searching method based on spectral image analysis and direct image alignment. S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop digests an image by its Fourier spectrum which provides a compact image hash value for fast candidate selection. The spectra also enable a convenient method of image alignment based on Fourier phase correction. The aligned images are further matched by an optical-flow process at the pixel granularity, which precisely estimates the frame dissimilarity against the turbulence caused by the perspective warping of random 3D scenes. According to our benchmark experiments on various datasets, S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop shows two orders of magnitude of less computing and memory overhead with comparable precision and recall over the prior DBoW method on a desktop PC. It is also much faster than the prior Fern method that has low computing complexity, and exhibits a much higher recall. While querying loops in 1000 runtime keyframes, S <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> Loop deployed onto a low-end STM32F4 MCU achieves 64fps.

Dynamic Detection Based on Planar Array Capacitance for Adhesive Defects of Thermal Protection Structure

The defect detection of thermal protective structures is of great significance in ensuring the safe flight of aircraft. In this paper, a dynamic measurement method based on the planar capacitance sensing principle is proposed to solve the problem of detecting defects in the adhesive layer of large-area thermal protection structures. This paper analyses the principle of capacitive sensing and establishes a planar array capacitive dynamic detection system. The dielectric distribution of the structural components to be measured is obtained at different locations by moving the array of electrode sensors regularly. To obtain the dielectric distribution information of the complete structure, the dielectric distribution at adjacent locations is spliced and fused. To achieve the precise location of the overlapping areas in the image to be fused, an image alignment method based on coordinates is adopted. The experimental results show that our proposed method can detect defects in the adhesive layer of large-area thermal protection structures and achieve complete visualization of the adhesive layer.

RI-Fusion: 3D Object Detection Using Enhanced Point Features With Range-Image Fusion for Autonomous Driving

The 3D object detection is becoming indispensable for environmental perception in autonomous driving. Light detection and ranging (LiDAR) point clouds often fail to distinguish objects with similar structures and are quite sparse for distant or small objects, thereby introducing false and missed detections. To address these issues, LiDAR is often fused with cameras due to the rich textural information provided by images. However, current fusion methods suffer the inefficient data representation and inaccurate alignment of heterogeneous features, leading to poor precision and low efficiency. To this end, we propose a plug-and-play module termed range-image fusion (RI-Fusion) to achieve an effective fusion of LiDAR and camera data, designed to be easily accessible by existing mainstream LiDAR-based algorithms. In this process, we design an image and point cloud alignment method by converting a point cloud into a compact range-view representation through a spherical coordinate transformation. The range image is then integrated with a corresponding camera image utilizing an attention mechanism. The original range image is then concatenated with fusion features to retain point cloud information, and the results are projected onto a spatial point cloud. Finally, the feature-enhanced point cloud can be input into a LiDAR-based 3D object detector. The results of validation experiments involving the KITTI 3D object detection benchmark showed that our proposed fusion method significantly enhanced multiple mainstream LiDAR-based 3D object detectors, PointPillars, SECOND, and Part <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{A}{^{2}}$ </tex-math></inline-formula> , improving the 3D mAP (mean Average Precision) by 3.61%, 2.98%, and 1.27%, respectively, particularly for small objects such as pedestrians and cyclists.

Semi-Direct Multimap SLAM System for Real-Time Sparse 3-D Map Reconstruction

The lack of a sufficient number of reliable corners in low-textured environments is a big challenge for classical visual simultaneous localization and mapping (SLAM), especially for point feature-based methods. Many other features (i.e., line and plane segments) are often combined with points to restore an environmental structure. However, using such features requires much computational time. This work focuses on the reliable and high-performance real-time operation of a SLAM system in low-textured scenarios. It proposes a semi-direct multi-map monocular SLAM system (SM-SLAM) that combines direct tracking and feature-based map maintenance with point features and line segments. The proposed system tracks non-keyframes based on a sparse image alignment method for fast tracking, extracts and matches point features and line segments in keyframes for high-quality environment structure and motion optimization. We present an extensive evaluation on two widely-used public datasets and some challenging real-world scenarios. Experimental results show that SM-SLAM can well reconstruct a sparse 3D map with geometrical structure information in 30-40 Hz. Notably, SM-SLAM shows an accuracy improvement of more than 20% than the Oriented FAST and Rotated BRIEF feature-based SLAM (ORB-SLAM) on some low-speed, small-range camera motion datasets and performs well in low-texture scenarios.

A multi-level image alignment method for aerial image and road-based geo-parcel data

Image alignment between aerial image and geo-parcel data is a meaningful and challenging task in remote sensing field. In this article, a deep learning framework based on multi-level progressive architecture focusing on aerial image and road based geo-parcel data alignment is proposed. Firstly, an image segmentation with U-Net model as a preprocessing work is applied to obtain the road binary image of aerial image, which benefits following image alignment by turning multi-modal image alignment into mono-modal image alignment. Afterwards, multi-scale deep features are extracted to take part in the proposed image alignment network. The proposed image alignment network consists of a global multiple homographies prediction module and a local flow map estimation module, forming a coarse-to-fine and global-to-local multi-level paradigm. Finally, synthetic image datasets are generated to test and verify the performance of proposed method. The experiments on the synthetic aerial image and road based geo-parcel data including Mnih dataset and Deep Globe road dataset demonstrate that the proposed method can align image pairs effectively, and the proposed method achieves a certain increase in matching performance by comparing with recent existing alignment methods qualitatively and quantitatively.

Scale-aware stereo direct visual odometry with online photometric calibration for agricultural environment

Vision-based localization and mapping for agricultural environment is challenging due to unstructured scene with less distinguishable landmarks and more repetitive textures, scale drift caused by wide and distant view, illumination variations and abrupt movements. To address these challenges, we propose a scale-aware stereo direct odometry with online photometric calibration for agricultural environment. We first propose a scale-aware stereo direct image alignment method to explicitly capture the scale at tracking stage. Then we optimize the scale and affine brightness parameters of stereo frames to keep the scale consistency in the sliding window. Furthermore, we incorporate online photometric calibration to resist illumination changes and exposure differences of image sequence. The proposed system achieves new state-of-the-art results on two public agricultural datasets, demonstrating its effectiveness in challenging agricultural environments.

Micro-expression spotting based on optical flow features

• The nose tip location-based image alignment method improves the performance of Micro-expression spotting. • The optical flow method can accurately capture micro-expression movements. • When a micro-expression occurs, a peak appears on the time-domain change curve. • Get complete micro-expression intervals after fusing local movements. Expression is the changes of facial organs due to facial muscle movement and an important way of human emotional interaction. Neurophysiological studies show that micro-expressions (MEs) are not controlled by subjective consciousness and reflect people's real emotions. That is why MEs have significant value in public security applications. This paper proposes an automatic ME spotting method of high accuracy and interpretability. Firstly, we design the nose tip location-based image alignment method to remove global displacement caused by head shaking. Secondly, according to the action unit definition in the face coding system (FACS), we select fourteen regions of interest (ROI) to capture subtle facial movements. The dense optical flow is introduced to estimate local movements and time-domain variations of the ROIs. Thirdly, we design a peak detection method on the time-domain variation curves to locate the movement intervals precisely. Lastly, we propose an overlapping index to measure the consistency of changes in different organs. Evaluation on the CAS(ME) 2 and SAMM Long Video database shows that our ME spotting method may achieve better accuracy with a relatively lower computation cost and can be applied to similar facial image processing applications.

Innovative Contactless Palmprint Recognition System Based on Dual-Camera Alignment

Recently, contactless bimodal palmprint recognition technology has attracted increased attention due to the COVID-19 pandemic. Many dual-camera-based sensors have been proposed to capture palm vein and palmprint images synchronously. However, translations between captured palmprint and palm vein images differ depending on the distance between the hand and the sensors. To address this issue, we designed a low-cost method to align the bimodal palm regions for current dual-camera systems. In this study, we first implemented a contactless palm image acquisition device with a dual-camera module and a single-point time of flight (TOF) ranging sensor. Using this device, we collected a dataset named DCPD under different distances and light source intensities from 271 different palms. Then, a bimodal palm image alignment method is proposed based on the imaging and ranging models. After the system model is calibrated, the translation between the visible light and infrared light palm regions can be estimated quickly based on the palm distance. Finally, we designed a convolutional neural network (CNN) to effectively extract the fine- and coarse-grained palm features. Compared to widely used existing methods, the proposed networks achieved the lowest equal error rate (EER) on the Tongji, IITD, and DCPD datasets, and the average time cost of the system to perform one-time identification is approximately 0.15 s. The experimental results indicate that the proposed methods achieved high efficiency and comparable accuracy. In addition, the system’s EER and rank-1 on the DCPD dataset were 0.304% and 98.66%, respectively.

Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI that typically works well only for small displacements; this points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while asymptotically superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g. rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, of aerial images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at \url{github.com/MIDA-group/globalign}.

Markerless Image Alignment Method for Pressure-Sensitive Paint Image.

We propose a markerless image alignment method for pressure-sensitive paint measurement data replacing the time-consuming conventional alignment method in which the black markers are placed on the model and are detected manually. In the proposed method, feature points are detected by a boundary detection method, in which the PSP boundary is detected using the Moore-Neighbor tracing algorithm. The performance of the proposed method is compared with the conventional method based on black markers, the difference of Gaussian (DoG) detector, and the Hessian corner detector. The results by the proposed method and the DoG detector are equivalent to each other. On the other hand, the performances of the image alignment using the black marker and the Hessian corner detector are slightly worse compared with the DoG and the proposed method. The computational cost of the proposed method is half of that of the DoG method. The proposed method is a promising for the image alignment in the PSP application in the viewpoint of the alignment precision and computational cost.

A Device-Independent, Shape Preserving Retinal Optical Coherence Tomography Image Alignment Method Applying TV-Unet for RPE Layer Detection

Retinal optical coherence tomography (OCT) images are widely used in diagnosis of ocular conditions. However, random shift and orientation changes of the retinal layers in OCT B-scans yield to appearance variations across the scans. These variations reduce the accuracy of the algorithms applied in the analysis of OCT images. In this study, we propose a preprocessing step to compensate these variations and align B-scans. At first, by incorporating total variation (TV) loss in the well-known Unet model, we propose a TV-Unet model to accurately detect the retinal pigment epithelium (RPE) layer in each B-scan. Then we use the detected RPE layer in the alignment method to form a curvature curve and a reference line. A novel window transferring-based alignment approach is applied to force the curve points to form a straight line, while preserving the shape and size of the pathological lesions. Since detection of RPE layer is a crucial step in the proposed alignment method, we utilized various datasets to train and test the TV-Unet and provided a multimodal, device-independent OCT image alignment method. The TV-Unet localizes the RPE layer in OCT images with low boundary error (maximum of 1.94pixels) and high Dice coefficient (minimum of 0.98). Quantitative and qualitative results indicated that the proposed method can efficiently detects the RPE layer and align OCT images while preserving the structure and size of the retinal lesions (biomarkers) in the OCT scans.