Multimodal Retinal Image Registration Research Articles

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute RBF of retinal microvasculature and OCTA can provide the structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg), which provides robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms existing methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.

Self-Supervised Rigid Registration for Multimodal Retinal Images.

The ability to accurately overlay one modality retinal image to another is critical in ophthalmology. Our previous framework achieved the state-of-the-art results for multimodal retinal image registration. However, it requires human-annotated labels due to the supervised approach of the previous work. In this paper, we propose a self-supervised multimodal retina registration method to alleviate the burdens of time and expense to prepare for training data, that is, aiming to automatically register multimodal retinal images without any human annotations. Specially, we focus on registering color fundus images with infrared reflectance and fluorescein angiography images, and compare registration results with several conventional and supervised and unsupervised deep learning methods. From the experimental results, the proposed self-supervised framework achieves a comparable accuracy comparing to the state-of-the-art supervised learning method in terms of registration accuracy and Dice coefficient.

Two-Step Registration on Multi-Modal Retinal Images via Deep Neural Networks.

Multi-modal retinal image registration plays an important role in the ophthalmological diagnosis process. The conventional methods lack robustness in aligning multi-modal images of various imaging qualities. Deep-learning methods have not been widely developed for this task, especially for the coarse-to-fine registration pipeline. To handle this task, we propose a two-step method based on deep convolutional networks, including a coarse alignment step and a fine alignment step. In the coarse alignment step, a global registration matrix is estimated by three sequentially connected networks for vessel segmentation, feature detection and description, and outlier rejection, respectively. In the fine alignment step, a deformable registration network is set up to find pixel-wise correspondence between a target image and a coarsely aligned image from the previous step to further improve the alignment accuracy. Particularly, an unsupervised learning framework is proposed to handle the difficulties of inconsistent modalities and lack of labeled training data for the fine alignment step. The proposed framework first changes multi-modal images into a same modality through modality transformers, and then adopts photometric consistency loss and smoothness loss to train the deformable registration network. The experimental results show that the proposed method achieves state-of-the-art results in Dice metrics and is more robust in challenging cases.

Perspective Distortion Correction for Multi-Modal Registration between Ultra-Widefield and Narrow-Angle Retinal Images.

Multi-modal retinal image registration between 2D Ultra-Widefield (UWF) and narrow-angle (NA) images has not been well-studied, since most existing methods mainly focus on NA image alignment. The stereographic projection model used in UWF imaging causes strong distortions in peripheral areas, which leads to inferior alignment quality. We propose a distortion correction method that remaps the UWF images based on estimated camera view points of NA images. In addition, we set up a CNN-based registration pipeline for UWF and NA images, which consists of the distortion correction method and three networks for vessel segmentation, feature detection and matching, and outlier rejection. Experimental results on our collected dataset shows the effectiveness of the proposed pipeline and the distortion correction method.

Robust Content-Adaptive Global Registration for Multimodal Retinal Images Using Weakly Supervised Deep-Learning Framework.

Multimodal retinal imaging plays an important role in ophthalmology. We propose a content-adaptive multimodal retinal image registration method in this paper that focuses on the globally coarse alignment and includes three weakly supervised neural networks for vessel segmentation, feature detection and description, and outlier rejection. We apply the proposed framework to register color fundus images with infrared reflectance and fluorescein angiography images, and compare it with several conventional and deep learning methods. Our proposed framework demonstrates a significant improvement in robustness and accuracy reflected by a higher success rate and Dice coefficient compared with other methods.

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration.

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.

Orientation-independent Feature Matching (OIFM) for Multimodal Retinal Image Registration

The analysis of fundus images in ophthalmology can be greatly facilitated by data integration of multimodal retinal images, which can be achieved via image registration based on the matching of keypoints represented by certain descriptors. However, the matching results offered by conventional feature descriptors may be substantially compromised due to their inconsistent estimates of the main orientation for keypoints in multimodal images. In this paper, we propose an orientation-independent feature matching (OIFM) method for better matching of feature points in multimodal retinal images. The keypoints detected in the images are firstly represented with a new circular neighborhood-based feature descriptor, allowing for the rotation of the neighborhood around the keypoints can be achieved by circularly shifting the elements in the descriptor. Then, the proposed feature descriptor is applied for the matching of keypoints, whose distance is measured after the compensation of the orientation discrepancy. The proposed OIFM method has two distinct characteristics. First, the feature descriptor no longer relies on the calculation of main orientations, and instead it is formulated by stacking the feature vectors of the keypoint neighborhood in a circular order with a conveniently assigned reference. Second, orientation independence is achieved during the matching stage where the feature points are aligned in a global manner, leading the OIFM method more robust to rotation and content variations among multimodal retinal images. Experimental results on a total of 160 pairs of multimodal retinal images show that the proposed OIFM method outperforms the conventional algorithms in terms of registration accuracy and robustness.

Study on Correlation Between Subjective and Objective Metrics for Multimodal Retinal Image Registration

Retinal imaging is crucial in diagnosing and treating retinal diseases, and multimodal retinal image registration constitutes a major advance in understanding retinal diseases. Despite the fact that many methods have been proposed for the registration task, the evaluation metrics for successful registration have not been thoroughly studied. In this article, we present a comprehensive overview of the existing evaluation metrics for multimodal retinal image registration, and compare the similarity between the subjective grade of ophthalmologists and various objective metrics. The Pearson's correlation coefficient and the corresponding confidence interval are used to evaluate metrics similarity. It is found that the binary and soft Dice coefficient on the segmented vessel can achieve the highest correlation with the subjective grades compared to other keypoint-supervised or unsupervised metrics. The paper established an objective metric that is highly correlated with the subjective evaluation of the ophthalmologists, which has never been studied before. The experimental results would build a connection between ophthalmology and image processing literature, and the findings may provide a good insight for researchers who investigate retinal image registration, retinal image segmentation and image domain transformation.

A-RANSAC: Adaptive random sample consensus method in multimodal retinal image registration

Abstract In this paper, an adaptive Random Sample Consensus (A-RANSAC) method is proposed for multimodal retinal image registration. In this method, the features of two various images from images taken with different modalities such as FA (Fluorescein angiography) and RF (Red free) are extracted using a modified version of Scale Invariant Feature Transform method (SIFT) called SAR-SIFT which is originally used for Synthetic Aperture Radar images. Then, the matching performance between these images is enhanced using the proposed A-RANSAC. In the A-RANSAC method, the threshold value is chosen so that the Root Mean Square Error (RMSE) and the number of removed matches are optimized simultaneously. The efficiency of the proposed method has been investigated in other modes such as high resolution and low-quality retinal image registration in addition to multimodal registration. The simulation results on several retinal image datasets show that the proposed method improves the precision matching by 9.89% and rate of success by 25% on the average compared to the SAR-SIFT method.

Multi-modal and multi-vendor retina image registration

Multi-modal retinal image registration is often required to utilize the complementary information from different retinal imaging modalities. However, a robust and accurate registration is still a challenge due to the modality-varied resolution, contrast, and luminosity. In this paper, a two step registration method is proposed to address this problem. Descriptor matching on mean phase images is used to globally register images in the first step. Deformable registration based on modality independent neighbourhood descriptor (MIND) method is followed to locally refine the registration result in the second step. The proposed method is extensively evaluated on color fundus images and scanning laser ophthalmoscope (SLO) images. Both qualitative and quantitative tests demonstrate improved registration using the proposed method compared to the state-of-the-art. The proposed method produces significantly and substantially larger mean Dice coefficients compared to other methods (p<0.001). It may facilitate the measurement of corresponding features from different retinal images, which can aid in assessing certain retinal diseases.

Robust point matching method for multimodal retinal image registration

In this paper, motivated by the problem of multimodal retinal image registration, we introduce and improve the robust registration framework based on partial intensity invariant feature descriptor (PIIFD), then present a registration framework based on speed up robust feature (SURF) detector, PIIFD and robust point matching, called SURF–PIIFD–RPM. Existing retinal image registration algorithms are unadaptable to any case, such as complex multimodal images, poor quality, and nonvascular images. Harris-PIIFD framework usually fails in correctly aligning color retinal images with other modalities when faced large content changes. Our proposed registration framework mainly solves the problem robustly. Firstly, SURF detector is useful to extract more repeatable and scale-invariant interest points than Harris. Secondly, a single Gaussian robust point matching model is based on the kernel method of reproducing kernel Hilbert space to estimate mapping function in the presence of outliers. Most importantly, our improved registration framework performs well even when confronted a large number of outliers in the initial correspondence set. Finally, multiple experiments on our 142 multimodal retinal image pairs demonstrate that our SURF–PIIFD–RPM outperforms existing algorithms, and it is quite robust to outliers.

Improving accuracy and efficiency of mutual information for multi-modal retinal image registration using adaptive probability density estimation

Mutual information (MI) is a popular similarity measure for performing image registration between different modalities. MI makes a statistical comparison between two images by computing the entropy from the probability distribution of the data. Therefore, to obtain an accurate registration it is important to have an accurate estimation of the true underlying probability distribution. Within the statistics literature, many methods have been proposed for finding the ‘optimal’ probability density, with the aim of improving the estimation by means of optimal histogram bin size selection. This provokes the common question of how many bins should actually be used when constructing a histogram. There is no definitive answer to this. This question itself has received little attention in the MI literature, and yet this issue is critical to the effectiveness of the algorithm. The purpose of this paper is to highlight this fundamental element of the MI algorithm. We present a comprehensive study that introduces methods from statistics literature and incorporates these for image registration. We demonstrate this work for registration of multi-modal retinal images: colour fundus photographs and scanning laser ophthalmoscope images. The registration of these modalities offers significant enhancement to early glaucoma detection, however traditional registration techniques fail to perform sufficiently well. We find that adaptive probability density estimation heavily impacts on registration accuracy and runtime, improving over traditional binning techniques.

An efficient approach for robust multimodal retinal image registration based on UR-SIFT features and PIIFD descriptors

Existing algorithms based on scale invariant feature transform (SIFT) and Harris corners such as edge-driven dual-bootstrap iterative closest point and Harris-partial intensity invariant feature descriptor (PIIFD) respectivley have been shown to be robust in registering multimodal retinal images. However, they fail to register color retinal images with other modalities in the presence of large content or scale changes. Moreover, the approaches need preprocessing operations such as image resizing to do well. This restricts the application of image registration for further analysis such as change detection and image fusion. Motivated by the need for efficient registration of multimodal retinal image pairs, this paper introduces a novel integrated approach which exploits features of uniform robust scale invariant feature transform (UR-SIFT) and PIIFD. The approach is robust against low content contrast of color images and large content, appearance, and scale changes between color and other retinal image modalities like the fluorescein angiography. Due to low efficiency of standard SIFT detector for multimodal images, the UR-SIFT algorithm extracts high stable and distinctive features in the full distribution of location and scale in images. Then, feature points are adequate and repeatable. Moreover, the PIIFD descriptor is symmetric to contrast, which makes it suitable for robust multimodal image registration. After the UR-SIFT feature extraction and the PIIFD descriptor generation in images, an initial cross-matching process is performed and followed by a mismatch elimination algorithm. Our dataset consists of 120 pairs of multimodal retinal images. Experiment results show the outperformance of the UR-SIFT-PIIFD over the Harris-PIIFD and similar algorithms in terms of efficiency and positional accuracy.

A Partial Intensity Invariant Feature Descriptor for Multimodal Retinal Image Registration

Detection of vascular bifurcations is a challenging task in multimodal retinal image registration. Existing algorithms based on bifurcations usually fail in correctly aligning poor quality retinal image pairs. To solve this problem, we propose a novel highly distinctive local feature descriptor named partial intensity invariant feature descriptor (PIIFD) and describe a robust automatic retinal image registration framework named Harris-PIIFD. PIIFD is invariant to image rotation, partially invariant to image intensity, affine transformation, and viewpoint/perspective change. Our Harris-PIIFD framework consists of four steps. First, corner points are used as control point candidates instead of bifurcations since corner points are sufficient and uniformly distributed across the image domain. Second, PIIFDs are extracted for all corner points, and a bilateral matching technique is applied to identify corresponding PIIFDs matches between image pairs. Third, incorrect matches are removed and inaccurate matches are refined. Finally, an adaptive transformation is used to register the image pairs. PIIFD is so distinctive that it can be correctly identified even in nonvascular areas. When tested on 168 pairs of multimodal retinal images, the Harris-PIIFD far outperforms existing algorithms in terms of robustness, accuracy, and computational efficiency.

Multimodal Retinal Image Registration for Optic Disk Segmentation

The analysis of the optic disk morphology with the means of the scanning laser tomography is an important step for glaucoma diagnosis. A method we developed for optic disk segmentation in images of the scanning laser tomograph is limited by noise, non-uniform illumination and presence of blood vessels. Inspired by recent medical research, we wanted to develop a tool for improving optic disk segmentation by registration of images of the scanning laser tomograph and color fundus photographs and by applying a method we developed for optic disk segmentation in color fundus photographs. The segmentation of the optic disk for glaucoma diagnosis in images of the scanning laser tomograph is based on morphological operations, detection of anatomical structures and active contours and has been described in a previous paper. The segmentation of the optic disk in the fundus photographs is based on nonlinear filtering, Canny edge detector and a modified Hough transform. The registration is based on mutual information using simulated annealing for finding maxima. The registration was successful 86.8% of the time when tested on 174 images. Results of the registration have shown a very low displacement error of a maximum of about 5 pixels. The correctness of the registration was manually evaluated by measuring distances between the real vessel borders and those from the registered image. We have developed a method for the registration of images of the scanning laser tomograph and fundus photographs. Our first experiments showed that the optic disk segmentation could be improved by fused information from both image modalities.