Multispectral Image Registration Research Articles

IFSrNet: Multi-Scale IFS Feature-Guided Registration Network Using Multispectral Image-to-Image Translation

Multispectral image registration is the process of aligning the spatial regions of two images with different distributions. One of the main challenges it faces is to resolve the severe inconsistencies between the reference and target images. This paper presents a novel multispectral image registration network, Multi-scale Intuitionistic Fuzzy Set Feature-guided Registration Network (IFSrNet), to address multispectral image registration. IFSrNet generates pseudo-infrared images from visible images using Cycle Generative Adversarial Network (CycleGAN), which is equipped with a multi-head attention module. An end-to-end registration network encodes the input multispectral images with intuitionistic fuzzification, which employs an improved feature descriptor—Intuitionistic Fuzzy Set–Scale-Invariant Feature Transform (IFS-SIFT)—to guide its operation. The results of the image registration will be presented in a direct output. For this task we have also designed specialised loss functions. The results of the experiment demonstrate that IFSrNet outperforms existing registration methods in the Visible–IR dataset. IFSrNet has the potential to be employed as a novel image-to-image translation paradigm.

Comprehensive Evaluation of Multispectral Image Registration Strategies in Heterogenous Agriculture Environment.

This article is focused on the comprehensive evaluation of alleyways to scale-invariant feature transform (SIFT) and random sample consensus (RANSAC) based multispectral (MS) image registration. In this paper, the idea is to extensively evaluate three such SIFT- and RANSAC-based registration approaches over a heterogenous mix containing Triticum aestivum crop and Raphanus raphanistrum weed. The first method is based on the application of a homography matrix, derived during the registration of MS images on spatial coordinates of individual annotations to achieve spatial realignment. The second method is based on the registration of binary masks derived from the ground truth of individual spectral channels. The third method is based on the registration of only the masked pixels of interest across the respective spectral channels. It was found that the MS image registration technique based on the registration of binary masks derived from the manually segmented images exhibited the highest accuracy, followed by the technique involving registration of masked pixels, and lastly, registration based on the spatial realignment of annotations. Among automatically segmented images, the technique based on the registration of automatically predicted mask instances exhibited higher accuracy than the technique based on the registration of masked pixels. In the ground truth images, the annotations performed through the near-infrared channel were found to have a higher accuracy, followed by green, blue, and red spectral channels. Among the automatically segmented images, the accuracy of the blue channel was observed to exhibit a higher accuracy, followed by the green, near-infrared, and red channels. At the individual instance level, the registration based on binary masks depicted the highest accuracy in the green channel, followed by the method based on the registration of masked pixels in the red channel, and lastly, the method based on the spatial realignment of annotations in the green channel. The instance detection of wild radish with YOLOv8l-seg was observed at a mAP@0.5 of 92.11% and a segmentation accuracy of 98% towards segmenting its binary mask instances.

Towards robust registration of heterogeneous multispectral UAV imagery: A two-stage approach for cotton leaf lesion grading

Multiple source images acquired from diverse sensors mounted on unmanned aerial vehicles (UAVs) offer valuable complementary information for ground vegetation analysis. However, accurately aligning heterogeneous UAV images poses challenges due to differences in geometry, intensity, and noise resulting from varying imaging principles. This paper presents a two-stage registration method aimed at fusing visible RGB and multispectral images for cotton leaf lesion grading. The coarse alignment stage utilizes Scale Invariant Feature Transform (SIFT), while the refined alignment stage employs a novel correlation coefficient-based template matching. The proposed method first employs the EfficientDet network to detect infected cotton leaves with lesions in RGB images. Subsequently, lesion leaves in multiple spectral imagery (red, green, red edge, and near-infrared bands) are located using the perspective transformation matrix derived from SIFT and the coordinates of lesion leaves in RGB images. Refined registration between RGB and multispectral imagery is achieved through template matching with the new correlation coefficient. The registered reflectance data from the different spectral bands and RGB components are utilized to classify pixels in each infected leaf into lesion, healthy, and soil parts. The lesion grade is determined based on the ratio of lesion pixels to the total corresponding leaf area. Experimental results, compared with manual assessment, demonstrate a lesion leaves detection model with a mAP@0.5 of 91.01% and a leaf lesion grading accuracy of 92.01%. These results validate the suitability of the proposed method for UAV RGB and multispectral image registration, enabling automated cotton leaf lesion grading.

PCNet: A structure similarity enhancement method for multispectral and multimodal image registration

Multispectral and multimodal images are of important usage in the field of multi-source visual information fusion. Due to the alternation or movement of image devices, the acquired multispectral and multimodal images are usually misaligned, and hence image registration is pre-requisite. Different from the registration of common images, the registration of multispectral or multimodal images is a challenging problem due to the nonlinear variation of intensity and gradient. To cope with this challenge, we propose the phase congruency network (PCNet) to enhance the structure similarity of multispectral or multimodal images. The images can then be aligned using the similarity-enhanced feature maps produced by the network. PCNet is constructed under the inspiration of the well-known phase congruency. The network embeds the phase congruency prior into two simple trainable layers and series of modified learnable Gabor kernels. Thanks to the prior knowledge, once trained, PCNet is applicable on a variety of multispectral and multimodal data such as flash/no-flash and RGB/NIR images without additional further tuning. The prior also makes the network lightweight. The trainable parameters of PCNet is 2400× and 1500×less than the deep-learning registration method deep homography network (DHN) and unsupervised deep homography network (UDHN), while its registration performance surpasses them. Experimental results validate that PCNet outperforms current state-of-the-art conventional multimodal registration algorithms. Besides, PCNet can act as a complementary part of the deep-learning registration methods, which significantly boosts their registration accuracy. On the Columbia imaging and vision laboratory (CAVE) multispectral dataset, the percentage of the number of images under 1 pixel average corner error (ACE) of UDHN is raised from 0.1% to 82.5% after the processing of PCNet.

PERFORMANCE EVALUATION OF LEARNING-BASED METHODS FOR MULTISPECTRAL SATELLITE IMAGE MATCHING

Abstract. Multispectral image registration is one of the most critical requirements to achieve reliable remote sensing goals such as change detection, image fusion, etc., due to providing complementary knowledge of the scene. On the one hand, this issue has always been a hot topic of research according to significant appearance differences, including geometric and nonlinear radiometric distortions. On the other hand, developing deep learning methods promises precise results in image processing and, in particular, image registration. It is no longer limited to low-level information structures, such as intensity and gradients. However, it is possible to provide more reliable results by extracting various high-level features and removing feature engineering. Therefore, we need extensive experiments in multispectral image registration to determine an efficient and robust method. To this end, this paper evaluates six well-known recently proposed learning-based feature descriptors, including LOFTR, TFeat, HardNet8, HardNet, SosNet, and HyNet, against geometric distortions within real multispectral images. Evaluations demonstrate the general superiority of the HardNet8 descriptor due to extracting high-level features within eight convolution layers.

CDFSL: Image Registration for Spaceborne Hyperspectral and Multispectral Data Having Large Spatial-Resolution Difference

Image registration aims to eliminate the geometric deviation between multi-source data with the same range, and to promote the collaborative application of data. In recent years, spaceborne hyperspectral (HS) and multispectral (MS) data have been widely used in Earth observation. However, the difference in the number of bands, spatial resolution, and spectral resolution puts forward higher requirements on the registration algorithm. The key to HS and MS image registration is to extract more common key points, weaken and eliminate the difference of radiation and spatial texture information to build superior descriptors, and achieve high-precision matching of key points. This paper introduces a new robust HS and MS registration method based on common deep feature subspaces. We first construct the common deep feature subspaces extraction network to extract consistent edge features and common subspace images of the image pair. Then, Harris algorithm is used to extract key points from consistent edge features between images, which reduces the impact of spatial resolution differences between images. Besides, the SIFT descriptor and subspace images are used to describe key points, which reduces the impact of radiation differences between images. Finally, Euclidean distance is used for the initial matching of key points, and the affine matrix is calculated after the outliers are eliminated, and image registration is performed. We perform experiments on spaceborne HS and MS datasets of different spatial resolutions and comparisons with state-of-the-art methods. Experimental results show that our method can obtain satisfactory registration results and is robust.

An Integrated Solution of UAV Push-Broom Hyperspectral System Based on Geometric Correction with MSI and Radiation Correction Considering Outdoor Illumination Variation

The unmanned aerial vehicle (UAV)-borne hyperspectral imaging system has the advantages of high spatial resolution, flexible operation, under-cloud flying, and easy cooperation with ground synchronous tests. Because this platform often flies under clouds, variations in solar illumination lead to irradiance inconsistency between different rows of hyperspectral images (HSIs). This inconsistency causes errors in radiation correction. In addition, due to the accuracy limitations of the GPS/inertial measurement unit (IMU) and irregular changes in flight platform speed and attitude, HSIs have deformation and drift, which is harmful to the geometric correction and stitching accuracy between flight strips. Consequently, radiation and geometric error limit further applications of large-scale hyperspectral data. To address the above problems, we proposed an integrated solution to acquire and correct UAV-borne hyperspectral images that consist of illumination data acquisition, radiance and geometric correction, HSI, multispectral image (MSI) registration, and multi-strip stitching. We presented an improved three-parameter empirical model based on the illumination correction factor, and it showed that the accuracy of radiation correction considering illumination variation improved, especially in some low signal-to-noise ratio (SNR) bands. In addition, the error of large-scale HSI stitching was controlled within one pixel.

On-Orbit Autonomous Geometric Calibration of Directional Polarimetric Camera

The Directional Polarimetric Camera (DPC) carried by the Chinese GaoFen-5-02 (GF-5-02) satellite has the ability for multiangle, multispectral, and polarization detection and will play an important role in the inversion of atmospheric aerosol and cloud characteristics. To ensure the validity of the DPC on-orbit multiangle and multispectral polarization data, high-precision image registration and geolocation are vital. High-precision geometric model parameters are a prerequisite for on-orbit image registration and geolocation. Therefore, on the basis of the multiangle imaging characteristics of DPC, an on-orbit autonomous geometric calibration method without ground reference data is proposed. The method includes three steps: (1) preprocessing the original image of the DPC and the satellite attitude and orbit parameters; (2) scale-invariant feature transform (SIFT) algorithm to match homologous points between multiangle images; (3) optimization of geometric model parameters on-orbit using least square theory. To verify the effectiveness of the on-orbit autonomous geometric calibration method, the image registration performance and relative geolocation accuracy before and after DPC on-orbit geometric calibration were evaluated and analyzed using the SIFT algorithm and the coastline crossing method (CCM). The results show that the on-orbit autonomous geometric calibration effectively improves the DPC image registration and relative geolocation accuracy. After on-orbit calibration, the multiangle image registration accuracy is better than 1.530 km, the multispectral image registration accuracy is better than 0.650 km, and the relative geolocation accuracy is better than 1.275 km, all reaching the subpixel level (<1.7 km).

Multispectral image registration based on an improved scale-invariant feature transform algorithm

There are often significant intensity variations between multispectral images, making automatic registration tasks difficult. Traditional feature matching methods, such as the scale-invariant feature transform (SIFT), are often sensitive to nonlinear variations of intensity between multispectral images. To solve this problem, an improved SIFT algorithm is introduced. First, the contrast limited adaptive histogram equalization algorithm is introduced in the feature extraction stage to improve the feature point extraction results. Then, the Sobel operator is used to enhance the main direction consistency between homologous feature point pairs. The experimental results suggest that the method can obtain reliable registration results on unmanned aerial vehicle multispectral images.

Robust Registration of Multispectral Satellite Images Based on Structural and Geometrical Similarity

Accurate registration of multispectral satellite images is a challenging task due to the significant and nonlinear radiometric differences between these data. To address this problem, this letter explores the strategy of geometrical similarity between triplets of feature points, and it is combined with the structural similarity between images in a feature-based image registration framework. The underlying principle is that the structural and geometrical similarities generally preserve across the images being registered. In this feature-based image registration framework, a set of control points (CPs) are first detected. Then, the geometric similarity between triplets of CPs is defined, followed by a ranking operation of these triplets of CPs. The highly ranked triplets are used to estimate a spatial transformation between images. Finally, initial matches obtained by a benchmark registration technique are refined by the estimated transformation. The experimental results demonstrate the great effectiveness of the proposed technique for registering multispectral satellite images.

Near Real-Time Automatic Sub-Pixel Registration of Panchromatic and Multispectral Images for Pan-Sharpening

This paper presents a near real-time automatic sub-pixel registration method of high-resolution panchromatic (PAN) and multispectral (MS) images using a graphics processing unit (GPU). In the first step, the method uses differential geo-registration to enable accurate geographic registration of PAN and MS images. Differential geo-registration normalizes PAN and MS images to the same direction and scale. There are also some residual misalignments due to the geometrical configuration of the acquisition instruments. These residual misalignments mean the PAN and MS images still have deviations after differential geo-registration. The second step is to use differential rectification with tiny facet primitive to eliminate possible residual misalignments. Differential rectification corrects the relative internal geometric distortion between PAN and MS images. The computational burden of these two steps is large, and traditional central processing unit (CPU) processing takes a long time. Due to the natural parallelism of the differential methods, these two steps are very suitable for mapping to a GPU for processing, to achieve near real-time processing while ensuring processing accuracy. This paper used GaoFen-6, GaoFen-7, ZiYuan3-02 and SuperView-1 satellite data to conduct an experiment. The experiment showed that our method’s processing accuracy is within 0.5 pixels. The automatic processing time of this method is about 2.5 s for 1 GB output data in the NVIDIA GeForce RTX 2080Ti, which can meet the near real-time processing requirements for most satellites. The method in this paper can quickly achieve high-precision registration of PAN and MS images. It is suitable for different scenes and different sensors. It is extremely robust to registration errors between PAN and MS.

Investigation and evaluation of algorithms for unmanned aerial vehicle multispectral image registration

Automatically registration of unmanned aerial vehicle (UAV) multispectral images is fundamental for subsequent applications. Although many studies exist for visible camera images and satellite multispectral image registration, studies for UAV are still rare. Under this context, this study firstly evaluates the performance of several widely used traditional methods (i.e., SIFT, SURF, ORB, and CFOG) for visible camera and satellite images in UAV multispectral image registration. This study further proposes an unsupervised and end-to-end deep learning network (i.e., DSIM) for multispectral image registration. An evident feature of DSIM is to regress the homography parameters with convolutional neural networks and to use the pyramid structural similarity loss to optimize the network. 1200 groups of UAV multispectral images acquired over three different sites in four months are used to comprehensively test the aforementioned five methods. Results show that CFOG achieves the highest correct matching rate in the test set, followed by DSIM and SIFT. Nevertheless, DSIM is more robust in images with weak or repeated texture than CFOG and SIFT. In addition, performance of CFOG and SIFT is closely related to the number of the found matching points. Based on the findings, we propose a multi-method ensemble strategy to combine CFOG, DSIM, and SIFT according to the number of the found matching points. This strategy outperforms the individual methods with a correct matching rate of 96.2%. Lower correct matching rate of CFOG + SIFT confirms that DSIM and traditional methods are very complementary in UAV multispectral image registrations.

ASRNet: Adversarial Segmentation and Registration Networks for Multispectral Fundus Images

Multispectral imaging (MSI) technique is often used to capture images of the fundus by illuminating it with different wavelengths of light. However, these images are taken at different points in time such that eyeball movements can cause misalignment between consecutive images. The multispectral image sequence reveals important information in the form of retinal and choroidal blood vessel maps, which can help ophthalmologists to analyze the morphology of these blood vessels in detail. This in turn can lead to a high diagnostic accuracy of several diseases. In this paper, we propose a novel semi-supervised end-to-end deep learning framework called “Adversarial Segmentation and Registration Nets” (ASRNet) for the simultaneous estimation of the blood vessel segmentation and the registration of multispectral images via an adversarial learning process. ASRNet consists of two subnetworks: (i) A segmentation module S that fulfills the blood vessel segmentation task, and (ii) A registration module R that estimates the spatial correspondence of an image pair. Based on the segmention-driven registration network, we train the segmentation network using a semi-supervised adversarial learning strategy. Our experimental results show that the proposed ASRNet can achieve state-of-the-art accuracy in segmentation and registration tasks performed with real MSI datasets.

Deep multispectral image registration network

Multispectral imaging (MSI) of the ocular fundus provides a sequence of narrow-band images to show the different depths in the retina and choroid. One challenge in analyzing MSI images comes from the image-to-image spatial misalignment, which occurs because the acquisition time of eye MSI images is commonly longer than the natural time scale of the eye's saccadic movement. It is necessary to align images because ophthalmologists usually overlay two of the images to analyze specific features when analyzing MSI images. In this paper, we propose a weakly supervised MSI image registration network, called MSI-R-NET, for multispectral fundus image registration. Compared to other deep-learning-based registration methods, MSI-R-NET utilizes the blood vessel segmentation label to provide spatial correspondence. In addition, we employ a feature equilibrium module to connect the aggregating layers better, and propose a multiresolution auto-context structure to adapt the registration task. In the testing stage, given a new pair of MSI images, the trained model can predict the pixelwise spatial correspondence without labeled blood vessel information. The experimental results demonstrate that the proposed segmentation-driven registration method is highly accurate.

RegiNet: Gradient guided multispectral image registration using convolutional neural networks

Multispectral image registration suffers from severe inconsistency between reference and target images. In this paper, we propose gradient guided multispectral image registration using convolutional neural networks, called RegiNet. We build an end-to-end network that directly produces the registration result from the input image pair. We use a gradient map of the reference image to guide the target image for registration. RegiNet first encodes the reference image and the gradient map of the target image separately, and then concatenates them to register the target image. For loss function, we use a structure loss to effectively capture gradient information from the reference image. Experimental results demonstrate that the proposed method successfully produces registration results as well as outperforms state-of-the-art ones in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

Boosting Structure Consistency for Multispectral and Multimodal Image Registration

Multispectral imaging plays a vital role in the area of computer vision and computational photography. As spectral band images can be misaligned due to imaging device movement or alternation, image registration is necessary to avoid spectral information distortion. The current registration measures specialized for multispectral data are typically robust yet complex, requiring excessive computation. The common measures such as sum of squared differences (SSD) and sum of absolute differences (SAD) are computationally efficient whereas they perform poorly on multispectral data. To cope with this challenge, we propose a structure consistency boosting (SCB) transform that aims at boosting the structural similarity of multispectral images. With SCB, the common measures can be employed for multispectral image registration. The SCB transform exploits the fact that inherent edge structures maintain relative saliency locally despite the nonlinear variation between band images. A statistical prior of the natural image, which is based on the gradient-intensity correlation, is explored to build a parametric form of SCB. Experimental results validate that the SCB transform outperforms current similarity enhancement algorithms, and performs better than the state-of-the-art multispectral registration measures. Thanks to the generality of the statistical prior, the SCB transform is also applicable to various multimodal data such as flash/no-flash images and medical images.

Multimodality registration for ocular multispectral images via co-embedding

Image registration of sequential multispectral images plays a vital role in retinal image analysis, since the appearance of ocular tissues significantly relates to the diagnosis, treatment, and evaluation of various diseases in ophthalmology. State-of-the-art multimodality image registration techniques greatly rely on mutual information between paired images to obtain their correspondence. However, it has been observed that mutual information-based image registration approaches suffer from inaccuracy especially when they are applied to small-sized images. Bearing this in mind, a novel groupwise registration approach is proposed by mapping the extracted features from multimodality images into the same latent space. To evaluate the proposed approach, the comparison experiments are conducted between state-of-the-art methods and the proposed approach. Experimental results demonstrate the superior accuracy of the proposed approach over the state-of-the-art techniques. Therefore, the proposed algorithm could be an invaluable tool for multimodality image registration applications.

An Integrated Approach to Registration and Fusion of Hyperspectral and Multispectral Images

Combining a hyperspectral (HS) image and a multispectral (MS) image - an example of image fusion - can result in a spatially and spectrally high-resolution image. Despite the plethora of fusion algorithms in remote sensing, a necessary prerequisite, namely registration, is mostly ignored. This limits their application to well-registered images from the same source. In this article, we propose and validate an integrated registration and fusion approach (code available at https://github.com/zhouyuanzxcv/Hyperspectral). The registration algorithm minimizes a least-squares (LSQ) objective function with the point spread function (PSF) incorporated together with a nonrigid freeform transformation applied to the HS image and a rigid transformation applied to the MS image. It can handle images with significant scale differences and spatial distortion. The fusion algorithm takes the full high-resolution HS image as an unknown in the objective function. Assuming that the pixels lie on a low-dimensional manifold invariant to local linear transformations from spectral degradation, the fusion optimization problem leads to a closed-form solution. The method was validated on the Pavia University, Salton Sea, and the Mississippi Gulfport datasets. When the proposed registration algorithm is compared to its rigid variant and two mutual information-based methods, it has the best accuracy for both the nonrigid simulated dataset and the real dataset, with an average error less than 0.15 pixels for nonrigid distortion of maximum 1 HS pixel. When the fusion algorithm is compared with current state-of-the-art algorithms, it has the best performance on images with registration errors as well as on simulations that do not consider registration effects.

Multi-Sensor Face Registration Based on Global and Local Structures

The work reported in this paper aims at utilizing the global geometrical relationship and local shape feature to register multi-spectral images for fusion-based face recognition. We first propose a multi-spectral face images registration method based on both global and local structures of feature point sets. In order to combine the global geometrical relationship and local shape feature in a new Student’s t Mixture probabilistic model framework. On the one hand, we use inner-distance shape context as the local shape descriptors of feature point sets. On the other hand, we formulate the feature point sets registration of the multi-spectral face images as the Student’s t Mixture probabilistic model estimation, and local shape descriptors are used to replace the mixing proportions of the prior Student’s t Mixture Model. Furthermore, in order to improve the anti-interference performance of face recognition techniques, a guided filtering and gradient preserving image fusion strategy is used to fuse the registered multi-spectral face image. It can make the multi-spectral fusion image hold more apparent details of the visible image and thermal radiation information of the infrared image. Subjective and objective registration experiments are conducted with manual selected landmarks and real multi-spectral face images. The qualitative and quantitative comparisons with the state-of-the-art methods demonstrate the accuracy and robustness of our proposed method in solving the multi-spectral face image registration problem.

Deep brain stimulation induced normalization of the human functional connectome in Parkinson's disease.

Neuroimaging has seen a paradigm shift away from a formal description of local activity patterns towards studying distributed brain networks. The recently defined framework of the 'human connectome' enables global analysis of parts of the brain and their interconnections. Deep brain stimulation (DBS) is an invasive therapy for patients with severe movement disorders aiming to retune abnormal brain network activity by local high frequency stimulation of the basal ganglia. Beyond clinical utility, DBS represents a powerful research platform to study functional connectomics and the modulation of distributed brain networks in the human brain. We acquired resting-state functional MRI in 20 patients with Parkinson's disease with subthalamic DBS switched on and off. An age-matched control cohort of 15 subjects was acquired from an open data repository. DBS lead placement in the subthalamic nucleus was localized using a state-of-the art pipeline that involved brain shift correction, multispectral image registration and use of a precise subcortical atlas. Based on a realistic 3D model of the electrode and surrounding anatomy, the amount of local impact of DBS was estimated using a finite element method approach. On a global level, average connectivity increases and decreases throughout the brain were estimated by contrasting on and off DBS scans on a voxel-wise graph comprising eight thousand nodes. Local impact of DBS on the motor subthalamic nucleus explained half the variance in global connectivity increases within the motor network (R = 0.711, P < 0.001). Moreover, local impact of DBS on the motor subthalamic nucleus could explain the degree to how much voxel-wise average brain connectivity normalized towards healthy controls (R = 0.713, P < 0.001). Finally, a network-based statistics analysis revealed that DBS attenuated specific couplings known to be pathological in Parkinson's disease. Namely, coupling between motor thalamus and motor cortex was increased while striatal coupling with cerebellum, external pallidum and subthalamic nucleus was decreased by DBS. Our results show that resting state functional MRI may be acquired in DBS on and off conditions on clinical MRI hardware and that data are useful to gain additional insight into how DBS modulates the functional connectome of the human brain. We demonstrate that effective DBS increases overall connectivity in the motor network, normalizes the network profile towards healthy controls and specifically strengthens thalamo-cortical connectivity while reducing striatal control over basal ganglia and cerebellar structures.