Rigid Registration of Multi-Modal Retinal Images Based on Intelligent Matching of Keypoint Pairs

There are different retinal vascular imaging modalities widely used in clinical practice to diagnose different retinal pathologies. The joint analysis of these multimodal images is of increasing interest since each of them provides common and complementary visual information. However, if we want to facilitate the comparison of two images, obtained with different techniques and containing the same retinal region of interest, it will be necessary to make a previous registration of both images. Here, we present a weakly supervised deep learning methodology for robust deformable registration of multimodal retinal images, which is applied to implement a method for the registration of fluorescein angiography (FA) and optical coherence tomography angiography (OCTA) images. This methodology is strongly inspired by VoxelMorph, a general unsupervised deep learning framework of the state of the art for deformable registration of unimodal medical images. The method was evaluated in a public dataset with 172 pairs of FA and superficial plexus OCTA images. The degree of alignment of the common information (blood vessels) and preservation of the non-common information (image background) in the transformed image were measured using the Dice coefficient (DC) and zero-normalized cross-correlation (ZNCC), respectively. The average values of the mentioned metrics, including the standard deviations, were DC = 0.72 ± 0.10 and ZNCC = 0.82 ± 0.04. The time required to obtain each pair of registered images was 0.12 s. These results outperform rigid and deformable registration methods with which our method was compared.

Retinal imaging employs various modalities, each providing distinct perspectives on ocular structures. However, the integration of information from these modalities, which often have differing resolutions, requires effective image registration techniques. Existing retinal image registration methods typically rely on rigid or affine transformations, which may not adequately address the complexities of multimodal retinal images. This study introduces a nonrigid fuzzy image registration approach designed to align optical coherence tomography (OCT) images with fundus images. The method employs a fuzzy inference system (FIS) that uses vessel locations as key features for registration. The FIS applies specific rules to map points from the source image to the reference image, facilitating accurate alignment. The proposed method achieved a mean absolute registration error of 44.57 ± 39.38 µm in the superior-inferior orientation and 11.46 ± 10.06 µm in the nasal-temporal orientation. These results underscore the method's precision in aligning multimodal retinal images. The nonrigid fuzzy image registration approach demonstrates robust and versatile performance in integrating multimodal retinal imaging data. Despite its straightforward implementation, the method effectively addresses the challenges of multimodal retinal image registration, providing a reliable tool for advanced ocular imaging analysis.

The registration of segmented retinal images is mainly used for the diagnosis of various diseases such as glaucoma, diabetes, and hypertension, etc. These retinal diseases depend on the retinal vessel structure. The fast and accurate registration of segmented retinal images helps to identify the changes in vessels and the diagnosis of the diseases. This paper presents a novel binary robust invariant scalable key point (BRISK) feature-based segmented retinal image registration approach. The BRISK framework is an efficient keypoint detection, description, and matching approach. The proposed approach contains three steps, namely, pre-processing, segmentation using matched filter based Gumbel pdf, and BRISK framework for registration of segmented source and target retinal images. The effectiveness of the proposed approach is demonstrated by evaluating the normalized cross-correlation of image pairs. Based on the experimental analysis, it has been observed that the performance of the proposed approach is better in both aspect, registration performance as well as computation time with respect to SURF and Harris partial intensity invariant feature descriptor based registration.

Existing algorithms based on scale invariant feature transform (SIFT) and Harris corners such as edge-driven dual-bootstrap iterative closest point and Harris-partial intensity invariant feature descriptor (PIIFD) respectivley have been shown to be robust in registering multimodal retinal images. However, they fail to register color retinal images with other modalities in the presence of large content or scale changes. Moreover, the approaches need preprocessing operations such as image resizing to do well. This restricts the application of image registration for further analysis such as change detection and image fusion. Motivated by the need for efficient registration of multimodal retinal image pairs, this paper introduces a novel integrated approach which exploits features of uniform robust scale invariant feature transform (UR-SIFT) and PIIFD. The approach is robust against low content contrast of color images and large content, appearance, and scale changes between color and other retinal image modalities like the fluorescein angiography. Due to low efficiency of standard SIFT detector for multimodal images, the UR-SIFT algorithm extracts high stable and distinctive features in the full distribution of location and scale in images. Then, feature points are adequate and repeatable. Moreover, the PIIFD descriptor is symmetric to contrast, which makes it suitable for robust multimodal image registration. After the UR-SIFT feature extraction and the PIIFD descriptor generation in images, an initial cross-matching process is performed and followed by a mismatch elimination algorithm. Our dataset consists of 120 pairs of multimodal retinal images. Experiment results show the outperformance of the UR-SIFT-PIIFD over the Harris-PIIFD and similar algorithms in terms of efficiency and positional accuracy.

A full migration BBO algorithm with enhanced population quality bounds for multimodal biomedical image registration

Registration of multi-modal retinal images is very significant to integrate information gained from different modalities for a reliable diagnosis of retinal diseases by ophthalmologists. However, accurate image registration is a challenging, we propose an algorithm for registration of summed-voxel projection images (SVPIs) with color fundus photographs (CFPs) based on local patch matching. SVPIs are evenly split into 16 local image blocks for extracting matching point pairs by searching local maximization of the similarity function. These matching point pairs are used for a coarse registration and then a search region of feature matching points is redefined for a more accurate registration. The performance of our registration algorithm is tested on a series of datasets including 3 normal eyes and 20 eyes with age-related macular degeneration. The experiment demonstrates that the proposed method can achieve accurate registration results (the average of root mean square error is 128μm).

Image registration is an essential task in image processing, where the final objective is to geometrically align two or more images. In remote sensing, this process allows comparing, fusing or analyzing data, specially when multi-modal images are used. In addition, multi-modal image registration becomes fairly challenging when the images have a significant difference in scale and resolution, together with local small image deformations. For this purpose, this paper presents a novel optical flow-based image registration network, named the FloU-Net, which tries to further exploit inter-sensor synergies by means of deep learning. The proposed method is able to extract spatial information from resolution differences and through an U-Net backbone generate an optical flow field estimation to accurately register small local deformations of multi-modal images in a self-supervised fashion. For instance, the registration between Sentinel-2 (S2) and Sentinel-3 (S3) optical data is not trivial, as there are considerable spectral-spatial differences among their sensors. In this case, the higher spatial resolution of S2 result in S2 data being a convenient reference to spatially improve S3 products, as well as those of the forthcoming Fluorescence Explorer (FLEX) mission, since image registration is the initial requirement to obtain higher data processing level products. To validate our method, we compare the proposed FloU-Net with other state-of-the-art techniques using 21 coupled S2/S3 optical images from different locations of interest across Europe. The comparison is performed through different performance measures. Results show that proposed FloU-Net can outperform the compared methods. The code and dataset are available in https://github.com/ibanezfd/FloU-Net.

Multi-modal image registration is a momentous technology in medical image processing and analysis. In order to improve the robustness and accuracy of multi-modal rigid image registration, a novel learning-based dissimilarity function is proposed in this paper. This novel dissimilarity function is based on measuring the dissimilarity between the joint intensity distribution of the testing image pair and the expected intensity distributions, which is learned from a registered image pair, with Bhattacharyya distances. Then, the aim of the registration process is to minimize the dissimilarity function. Eight hundred randomized CT - T1 registrations were performed and evaluated by the Retrospective Image Registration Evaluation (RIRE) project. The experimental results demonstrate that the proposed method can achieve higher robustness and accuracy, as compared with a closely related approach and a state-of-the-art method.

Multi-modal image registration is a challenging problem in medical imaging. The goal is to align anatomically identical structures; however, their appearance in images acquired with different imaging devices, such as CT or MR, may be very different. Registration algorithms generally deform one image, the floating image, such that it matches with a second, the reference image, by maximizing some similarity score between the deformed and the reference image. Instead of using a universal, but a priori fixed similarity criterion such as mutual information, we propose learning a similarity measure in a discriminative manner such that the reference and correctly deformed floating images receive high similarity scores. To this end, we develop an algorithm derived from max-margin structured output learning, and employ the learned similarity measure within a standard rigid registration algorithm. Compared to other approaches, our method adapts to the specific registration problem at hand and exploits correlations between neighboring pixels in the reference and the floating image. Empirical evaluation on CT-MR/PET-MR rigid registration tasks demonstrates that our approach yields robust performance and outperforms the state of the art methods for multi-modal medical image registration.

Multi-modal image registration is a challenging problem in medical imaging. The goal is to align anatomically identical structures; however, their appearance in images acquired with different imaging devices, such as CT or MR, may be very different. Registration algorithms generally deform one image, the floating image, such that it matches with a second, the reference image, by maximizing some similarity score between the deformed and the reference image. Instead of using a universal, but a priori fixed similarity criterion such as mutual information, we propose learning a similarity measure in a discriminative manner such that the reference and correctly deformed floating images receive high similarity scores. To this end, we develop an algorithm derived from max-margin structured output learning, and employ the learned similarity measure within a standard rigid registration algorithm. Compared to other approaches, our method adapts to the specific registration problem at hand and exploits correlations between neighboring pixels in the reference and the floating image. Empirical evaluation on CT-MR/PET-MR rigid registration tasks demonstrates that our approach yields robust performance and outperforms the state of the art methods for multi-modal medical image registration.

Multimodal retinal images (RI) are extensively used for analysing various eye diseases and conditions such as myopia and diabetic retinopathy. The incorporation of either two or more RI modalities provides complementary structure information in the presence of non-uniform illumination and low-contrast homogeneous regions. It also presents significant challenges for retinal image registration (RIR). This paper investigates how the Expectation Maximization for Principal Component Analysis with Mutual Information (EMPCA-MI) algorithm can effectively achieve multimodal RIR. This iterative hybrid-based similarity measure combines spatial features with mutual information to provide enhanced registration without recourse to either segmentation or feature extraction. Experimental results for clinical multimodal RI datasets comprising colour fundus and scanning laser ophthalmoscope images confirm EMPCA-MI is able to consistently afford superior numerical and qualitative registration performance compared with existing RIR techniques, such as the bifurcation structures method.

This paper presents a novel feature-based method that is built upon a convolutional neural network (CNN) to learn the deep representation for multimodal retinal image registration. We coined the algorithm deep step patterns, in short DeepSPa. Most existing deep learning based methods require a set of manually labeled training data with known corresponding spatial transformations, which limits the size of training datasets. By contrast, our method is fully automatic and scale well to different image modalities with no human intervention. We generate feature classes from simple step patterns within patches of connecting edges formed by vascular junctions in multiple retinal imaging modalities. We leverage CNN to learn and optimize the input patches to be used for image registration. Spatial transformations are estimated based on the output possibility of the fully connected layer of CNN for a pair of images. One of the key advantages of the proposed algorithm is its robustness to non-linear intensity changes, which widely exist on retinal images due to the difference of acquisition modalities. We validate our algorithm on extensive challenging datasets comprising poor quality multimodal retinal images which are adversely affected by pathologies (diseases), speckle noise and low resolutions. The experimental results demonstrate the robustness and accuracy over state-of-the-art multimodal image registration algorithms.

Retinal imaging is crucial in diagnosing and treating retinal diseases, and multimodal retinal image registration constitutes a major advance in understanding retinal diseases. Despite the fact that many methods have been proposed for the registration task, the evaluation metrics for successful registration have not been thoroughly studied. In this article, we present a comprehensive overview of the existing evaluation metrics for multimodal retinal image registration, and compare the similarity between the subjective grade of ophthalmologists and various objective metrics. The Pearson's correlation coefficient and the corresponding confidence interval are used to evaluate metrics similarity. It is found that the binary and soft Dice coefficient on the segmented vessel can achieve the highest correlation with the subjective grades compared to other keypoint-supervised or unsupervised metrics. The paper established an objective metric that is highly correlated with the subjective evaluation of the ophthalmologists, which has never been studied before. The experimental results would build a connection between ophthalmology and image processing literature, and the findings may provide a good insight for researchers who investigate retinal image registration, retinal image segmentation and image domain transformation.

Deformable multimodal image registration plays a key role in medical image analysis. It remains a challenge to find accurate dense correspondences between multimodal images due to the significant intensity distortion and the large deformation. macJNet is proposed to align the multimodal medical images, which is a weakly-supervised multimodal image deformable registration method using a joint learning framework and multi-sampling cascaded modality independent neighborhood descriptor (macMIND). The joint learning framework consists of a multimodal image registration network and two segmentation networks. The proposed macMIND is a modality-independent image structure descriptor to provide dense correspondence for registration, which incorporates multi-orientation and multi-scale sampling patterns to build self-similarity context. It greatly enhances the representation ability of cross-modal features in the registration network. The semi-supervised segmentation networks generate anatomical labels to provide semantics correspondence for registration, and the registration network helps to improve the performance of multimodal image segmentation by providing the consistency of anatomical labels. 3D CT-MR liver image dataset with 118 samples is built for evaluation, and comprehensive experiments have been conducted to demonstrate that macJNet achieves superior performance over state-of-the-art multi-modality medical image registration methods.

Images captured using different modalities usually have significant variations in their intensities, which makes it difficult to reveal their internal structural similarities and achieve accurate registration. Most conventional feature-based image registration techniques are fast and efficient, but they cannot be used directly for the registration of multi-modal images because of these intensity variations. This paper introduces the theory of manifold learning to transform the original images into mono-modal modalities, which is a feature-based method that is applicable to multi-modal image registration. Subsequently, scale-invariant feature transform is used to detect highly distinctive local descriptors and matches between corresponding images, and a point-based registration is executed. The algorithm was tested with T1- and T2-weighted magnetic resonance (MR) images obtained from BrainWeb. Both qualitative and quantitative evaluations of the method were performed and the results compared with those produced previously. The experiments showed that feature point matching after manifold learning achieved more accurate results than did the similarity measure for multi-modal image registration. This study provides a new manifold-based feature point matching method for multi-modal medical image registration, especially for MR images. The proposed method performs better than do conventional intensity-based techniques in terms of its registration accuracy and is suitable for clinical procedures.