Pairs Of Retinal Images Research Articles

Heatmap analysis for artificial intelligence explainability in diabetic retinopathy detection: illuminating the rationale of deep learning decisions.

The opaqueness of artificial intelligence (AI) algorithms decision processes limit their application in healthcare. Our objective was to explore discrepancies in heatmaps originated from slightly different retinal images from the same eyes of individuals with diabetes, to gain insights into the deep learning (DL) decision process. Pairs of retinal images from the same eyes of individuals with diabetes, composed of images obtained before and after pupil dilation, underwent automatic analysis by a convolutional neural network for the presence of diabetic retinopathy (DR), output being a score ranging from 0 to 1. Gradient-based Class Activation Maps (GradCam) allowed visualization of activated areas. Pairs of images with discordant DL scores or outputs within the pair were objectively compared to the concordant pairs, regarding the sum of activations of Class Activation Mapping (CAM), the number of activated areas, and DL score differences. Heatmaps of discordant pairs were also qualitatively assessed. Algorithmic performance for the detection of DR attained 89.8% sensitivity, 96.3% specificity and area under the receiver operating characteristic (ROC) curve of 0.95. Out of 210 comparable pairs of images, 20 eyes and 10 eyes were considered discordant according to DL score difference and regarding DL output, respectively. Comparison of concordant versus discordant groups showed statistically significant differences for all objective variables. Qualitative analysis pointed to subtle differences in image quality within discordant pairs. The successfully established relationship among objective parameters extracted from heatmaps and DL output discrepancies reinforces the role of heatmaps for DL explainability, fostering acceptance of DL systems for clinical use.

Robust image registration for analysis of multisource eye fundus images

This study underscores the crucial role of image preprocessing in enhancing the outcomes of multimodal image registration tasks using scale-invariant feature selection. The primary focus is on registering two types of retinal images, assessing a methodology’s performance on a set of retinal image pairs, including those with and without microaneurysms. Each pair comprises a color optical image and a gray-level fluorescein image, presenting distinct characteristics and captured under varying conditions. The SIFT methodology, encompassing five stages, with preprocessing as the initial and pivotal stage, is employed for image registration. Out of 35 test retina image pairs, 33 (94.28%) were successfully registered, with the inability to extract features hindering automatic registration in the remaining pairs. Among the registered pairs, 42.42% were retinal images without microaneurysms, and 57.57% had microaneurysms. Instead of simultaneous registration of all channels, independent registration of preprocessed images in each channel proved more effective. The study concludes with an analysis of the fifth registration’s resulting image to detect abnormalities or pathologies, highlighting the challenges encountered in registering blue channel images due to high intrinsic noise.

Changed Detection Based on Patch Robust Principal Component Analysis

Change detection on retinal fundus image pairs mainly seeks to compare the important differences between a pair of images obtained at two different time points such as in anatomical structures or lesions. Illumination variation usually challenges the change detection methods in many cases. Robust principal component analysis (RPCA) takes intensity normalization and linear interpolation to greatly reduce the illumination variation between the continuous frames and then decomposes the image matrix to obtain the robust background model. The matrix-RPCA can obtain clear change regions, but when there are local bright spots on the image, the background model is vulnerable to illumination, and the change detection results are inaccurate. In this paper, a patch-based RPCA (P-RPCA) is proposed to detect the change of fundus image pairs, where a pair of fundus images is normalized and linearly interpolated to expand a low-rank image sequence; then, images are divided into many patches to obtain an image-patch matrix, and finally, the change regions are obtained by the low-rank decomposition. The proposed method is validated on a set of large lesion image pairs in clinical data. The area under curve (AUC) and mean average precision (mAP) of the method proposed in this paper are 0.9832 and 0.8641, respectively. For a group of small lesion image pairs with obvious local illumination changes in clinical data, the AUC and mAP obtained by the P-RPCA method are 0.9893 and 0.9401, respectively. The results show that the P-RPCA method is more robust to local illumination changes than the RPCA method, and has stronger performance in change detection than the RPCA method.

Fusion of multi-source retinal fundus images via automatic registration for clinical diagnosis

Diabetic retinopathy, age-related macular degeneration and glaucoma, are the leading causes of visual impairment or blindness of the population across different ages. Retinal fundus imaging is a clinically regular tool for the diagnosis of retinal diseases. In the interest of having a comprehensive understanding of the fundus condition, it is valuable to leverage multiple fundus images from different modalities. However, a direct fusion of the multi-source fundus images eases to mis-align the physiological structure or spatial position due to possible eyeball rotations or head movements. The problem turns out to be more severe if the images were corrupted by ill conditions on eyes, such as micro-bleeding and plaques. To tackle this problem, we propose a multi-source registration model for retinal fundus images. Our proposed method considers multiple correspondences and dual structural constraints during the registration process. The method firstly selects adequate feature points by an adjustable threshold selection strategy. Then a feature-guided correspondence estimation model is established to build complementary features. Finally, their spatial transformation is built by using mean shift evolution. The evolution is guided by Tikhonov regularization on dual geometric structures. It overcomes the mess of mean shift vector field and mitigating the ill-posed displacement in field recovery. We have conducted our method on the collected 220 multi-source retinal fundus image pairs, which involve minor and larger displacement or severe retinopathy lesions, as well as additive different intensities of Gaussian noises. Extensive experiments demonstrate that the proposed method consistently outperforms seven feature-based methods.

Double-path parallel convolutional neural network for removing speckle noise in different types of OCT images

Speckle noises widely exist in optical coherence tomography (OCT) images. We propose an improved double-path parallel convolutional neural network (called DPNet) to reduce speckles. We increase the network width to replace the network depth to extract deeper information from the original OCT images. In addition, we use dilated convolution and residual learning to increase the learning ability of our DPNet. We use 100 pairs of human retinal OCT images as the training dataset. Then we test the DPNet model for denoising speckles on four different types of OCT images, mainly including human retinal OCT images, skin OCT images, colon crypt OCT images, and quail embryo OCT images. We compare the DPNet model with the adaptive complex diffusion method, the curvelet shrinkage method, the shearlet-based total variation method, and the OCTNet method. We qualitatively and quantitatively evaluate these methods in terms of image smoothness, structural information protection, and edge clarity. Our experimental results prove the performance of the DPNet model, and it allows us to batch and quickly process different types of poor-quality OCT images without any parameter fine-tuning under a time-constrained situation.

Retinal image mosaicking using scale-invariant feature transformation feature descriptors and Voronoi diagram.

Purpose: Peripheral retinal lesions substantially increase the risk of diabetic retinopathy and retinopathy of prematurity. The peripheral changes can be visualized in wide field imaging, which is obtained by combining multiple images with an overlapping field of view using mosaicking methods. However, a robust and accurate registration of mosaicking techniques for normal angle fundus cameras is still a challenge due to the random selection of matching points and execution time. We propose a method of retinal image mosaicking based on scale-invariant feature transformation (SIFT) feature descriptor and Voronoi diagram. Approach: In our method, the SIFT algorithm is used to describe local features in the input images. Then the input images are subdivided into regions based on the Voronoi method. Each pair of Voronoi regions is matched by the method zero mean normalized cross correlation. After matching, the retinal images are mapped into the same coordinate system to form a mosaic image. The success rate and the mean registration error (RE) of our method were compared with those of other state-of-the-art methods for the P category of the fundus image registration database. Results: Experimental results show that the proposed method accurately registered 42% of retinal image pairs with a mean RE of 3.040pixels, while a lower success rate was observed in the other four state-of-the-art retinal image registration methods GDB-ICP (33%), Harris-PIIFD (0%), HM-2016 (0%), and HM-2017 (2%). Conclusions: The proposed method outperforms state-of-the-art methods in terms of quality and running time and reduces the computational complexity.

Registration of fluorescein angiography and optical coherence tomography images of curved retina via scanning laser ophthalmoscopy photographs.

Accurate and automatic registration of multimodal retinal images such as fluorescein angiography (FA) and optical coherence tomography (OCT) enables utilization of supplementary information. FA is a gold standard imaging modality that depicts neurovascular structure of retina and is used for diagnosing neurovascular-related diseases such as diabetic retinopathy (DR). Unlike FA, OCT is non-invasive retinal imaging modality that provides cross-sectional data of retina. Due to differences in contrast, resolution and brightness of multimodal retinal images, the images resulted from vessel extraction of image pairs are not exactly the same. Also, prevalent feature detection, extraction and matching schemes do not result in perfect matches. In addition, the relationships between retinal image pairs are usually modeled by affine transformation, which cannot generate accurate alignments due to the non-planar retina surface. In this paper, a precise registration scheme is proposed to align FA and OCT images via scanning laser ophthalmoscopy (SLO) photographs as intermediate images. For this purpose, first a retinal vessel segmentation is applied to extract main blood vessels from the FA and SLO images. Next, a novel global registration is proposed based on the Gaussian model for curved surface of retina. For doing so, first a global rigid transformation is applied to FA vessel-map image using a new feature-based method to align it with SLO vessel-map photograph, in a way that outlier matched features resulted from not-perfect vessel segmentation are completely eliminated. After that, the transformed image is globally registered again considering Gaussian model for curved surface of retina to improve the precision of the previous step. Eventually a local non-rigid transformation is exploited to register two images perfectly. The experimental results indicate the presented scheme is more precise compared to other registration methods.

Orientation-independent Feature Matching (OIFM) for Multimodal Retinal Image Registration

The analysis of fundus images in ophthalmology can be greatly facilitated by data integration of multimodal retinal images, which can be achieved via image registration based on the matching of keypoints represented by certain descriptors. However, the matching results offered by conventional feature descriptors may be substantially compromised due to their inconsistent estimates of the main orientation for keypoints in multimodal images. In this paper, we propose an orientation-independent feature matching (OIFM) method for better matching of feature points in multimodal retinal images. The keypoints detected in the images are firstly represented with a new circular neighborhood-based feature descriptor, allowing for the rotation of the neighborhood around the keypoints can be achieved by circularly shifting the elements in the descriptor. Then, the proposed feature descriptor is applied for the matching of keypoints, whose distance is measured after the compensation of the orientation discrepancy. The proposed OIFM method has two distinct characteristics. First, the feature descriptor no longer relies on the calculation of main orientations, and instead it is formulated by stacking the feature vectors of the keypoint neighborhood in a circular order with a conveniently assigned reference. Second, orientation independence is achieved during the matching stage where the feature points are aligned in a global manner, leading the OIFM method more robust to rotation and content variations among multimodal retinal images. Experimental results on a total of 160 pairs of multimodal retinal images show that the proposed OIFM method outperforms the conventional algorithms in terms of registration accuracy and robustness.

Image Registration Based on Camera Calibration for Dual-Wavelength Retinal Oximetry

Retinal images at 570nm and 600nm are simultaneously captured by retinal oximetry. However, there exists misalignment between the two images, which should be adjusted before they are used to analyze hemoglobin oxygen saturation (SO2). Having a robust, accurate and fast registration scheme can ensure accuracy and speed of SO2 calculation. However, the existing retinal image registration methods cannot meet demand for clinical application of retinal oximetry. For example, errors in the vessel segmentation from each image cause corresponding errors in the registration, point matches may become unreliable when the predefined matching cost is corrupted by pathological retinal images. And it is always a time-consuming task. In this paper, we first analyzed the misalignment reason between dual-wavelength images, and concluded that there only exists the rotation and translation deviation between the two images caused by a mounting deviation of CCD cameras. The parameters were only related to system itself. Based on this, we proposed a registration method based on camera calibration that benefits from feature extraction and affine transformation matrix calculation. The parameters obtained by the proposed method were tested on 80 pairs of retinal images of 5 healthy volunteers and the results showed that the proposed method can successfully register the dual-wavelength images. The proposed registration method can significantly improve SO2 calculation speed while ensuring accuracy.

Feature guided Gaussian mixture model with semi-supervised EM and local geometric constraint for retinal image registration

Retinal image registration, which can be formulated by matching two sets of sparse features extracted from two observed retinal images, is a crucial step in the diagnosis and treatment of various eye diseases. Existing methods suffer from missing true correspondences or do not fully consider local appearance information, which causes difficulty in matching low-quality retinal images due to insufficient reliable features. In addition, the relationships between retinal image pairs are usually modeled by linear transformation, such as affine transformation, which cannot generate accurate alignments in large viewpoint changes due to the nonplanar eyeball surface. To address these issues, a feature guided Gaussian mixture model (GMM) is proposed for the non-rigid registration of retinal images. We formulate the problem as an estimation of a feature guided mixture of densities: a GMM is fitted to one point set in which the centers of the Gaussian densities characterized by spatial positions associated with local appearance descriptors are constrained to coincide with the other point set. The problem is solved under a maximum-likelihood framework, and semi-supervised expectation-maximization is used to iteratively estimate the feature correspondence and spatial transformation, which is initialized by a set of confidential feature matches obtained previously. Non-rigid transformation is specified in a reproducing kernel Hilbert space, and a local geometric constraint is imposed to establish the transformation estimation for obtaining a meaningful solution. A fast implementation based on sparse approximation is also provided and reduces the time complexity from cubic to quadratic. Moreover, we use the edge map, which can extract more reliable features, as a uniform representation of retinal images. Experimental results on publicly available retinal images show that our approach is robust in different registration tasks and outperforms several competing approaches, especially when data is severely degraded.

FIRE: Fundus Image Registration dataset

Purpose: Retinal image registration is a useful tool for medical professionals. However, performance evaluation of registration methods has not been consistently assessed in the literature. To address that, a dataset comprised of retinal image pairs annotated with ground truth and an evaluation protocol for registration methods is proposed.Methods: The dataset is comprised by 134 retinal fundus image pairs. These pairs are classified into three categories, according to characteristics that are relevant to indicative registration applications. Such characteristics are the degree of overlap between images and the presence/absence of anatomical differences. Ground truth in the form of corresponding image points and a protocol to evaluate registration performance are provided.Results: The proposed protocol is shown to enable quantitative and comparative evaluation of retinal registration methods under a variety of conditions.Conclusion: This work enables the fair comparison of retinal registration methods. It also helps researchers to select the registration method that is most appropriate given a specific target use.

Retinal biometrics based on Iterative Closest Point algorithm.

The pattern of blood vessels in the eye is unique to each person because it rarely changes over time. Therefore, it is well known that retinal blood vessels are useful for biometrics. This paper describes a biometrics method using the Jaccard similarity coefficient (JSC) based on blood vessel regions in retinal image pairs. The retinal image pairs were rough matched by the center of their optic discs. Moreover, the image pairs were aligned using the Iterative Closest Point algorithm based on detailed blood vessel skeletons. For registration, perspective transform was applied to the retinal images. Finally, the pairs were classified as either correct or incorrect using the JSC of the blood vessel region in the image pairs. The proposed method was applied to temporal retinal images, which were obtained in 2009 (695 images) and 2013 (87 images). The 87 images acquired in 2013 were all from persons already examined in 2009. The accuracy of the proposed method reached 100%.

Feature-Based Retinal Image Registration Using D-Saddle Feature.

Retinal image registration is important to assist diagnosis and monitor retinal diseases, such as diabetic retinopathy and glaucoma. However, registering retinal images for various registration applications requires the detection and distribution of feature points on the low-quality region that consists of vessels of varying contrast and sizes. A recent feature detector known as Saddle detects feature points on vessels that are poorly distributed and densely positioned on strong contrast vessels. Therefore, we propose a multiresolution difference of Gaussian pyramid with Saddle detector (D-Saddle) to detect feature points on the low-quality region that consists of vessels with varying contrast and sizes. D-Saddle is tested on Fundus Image Registration (FIRE) Dataset that consists of 134 retinal image pairs. Experimental results show that D-Saddle successfully registered 43% of retinal image pairs with average registration accuracy of 2.329 pixels while a lower success rate is observed in other four state-of-the-art retinal image registration methods GDB-ICP (28%), Harris-PIIFD (4%), H-M (16%), and Saddle (16%). Furthermore, the registration accuracy of D-Saddle has the weakest correlation (Spearman) with the intensity uniformity metric among all methods. Finally, the paired t-test shows that D-Saddle significantly improved the overall registration accuracy of the original Saddle.

Retinal image registration under the assumption of a spherical eye

We propose a method for registering a pair of retinal images. The proposed approach employs point correspondences and assumes that the human eye has a spherical shape. The image registration problem is formulated as a 3D pose estimation problem, solved by estimating the rigid transformation that relates the views from which the two images were acquired. Given this estimate, each image can be warped upon the other so that pixels with the same coordinates image the same retinal point. Extensive experimental evaluation shows improved accuracy over state of the art methods, as well as robustness to noise and spurious keypoint matches. Experiments also indicate the method's applicability to the comparative analysis of images from different examinations that may exhibit changes and its applicability to diagnostic support.

3D surface reconstruction of retinal vascular structures

We propose in this paper, a three-dimensional surface reconstruction of a retinal vascular network from a pair of 2D retinal images. Our approach attempts to address the above challenges by incorporating an epipolar geometry estimation and adaptive surface modelling in a 3D reconstruction, using three steps: segmentation, 3D skeleton reconstruction and 3D surface modelling of vascular structures. The intrinsic calibration matrices are found via the solution of simplified Kruppa equations. A simple essential matrix based on a self-calibration method has been used for the fundus camera-eye system. The used method has eventually produced vessel surfaces that could be fit for various applications, such as applications for computational fluid dynamics simulations and applications for real-time virtual interventional.

Robust point matching method for multimodal retinal image registration

In this paper, motivated by the problem of multimodal retinal image registration, we introduce and improve the robust registration framework based on partial intensity invariant feature descriptor (PIIFD), then present a registration framework based on speed up robust feature (SURF) detector, PIIFD and robust point matching, called SURF–PIIFD–RPM. Existing retinal image registration algorithms are unadaptable to any case, such as complex multimodal images, poor quality, and nonvascular images. Harris-PIIFD framework usually fails in correctly aligning color retinal images with other modalities when faced large content changes. Our proposed registration framework mainly solves the problem robustly. Firstly, SURF detector is useful to extract more repeatable and scale-invariant interest points than Harris. Secondly, a single Gaussian robust point matching model is based on the kernel method of reproducing kernel Hilbert space to estimate mapping function in the presence of outliers. Most importantly, our improved registration framework performs well even when confronted a large number of outliers in the initial correspondence set. Finally, multiple experiments on our 142 multimodal retinal image pairs demonstrate that our SURF–PIIFD–RPM outperforms existing algorithms, and it is quite robust to outliers.

Solving the correspondence problem in stereo vision by internal simulation

We present a computational model for object matching in a pair of stereo images based on internal sensorimotor simulation. In our study, we use pairs of retinal images, i.e. the resolution is higher towards the image center and low in the periphery, which stem from two cameras, each one mounted on a pan–tilt unit (PTU). The internal simulation is driven by two internal models: a saccade controller (SC) which generates a fixation movement to a certain point in either image, and a visual forward model (VFM) that models the effect on camera movements (by the PTU) onto the image. The SC takes as sensory input the current position of a salient point (in image coordinates) and generates a motor command that would lead to the fixation of that point. The VFM takes as sensory input a current camera image and a motor command, i.e. a saccade, and generates an image that appears as if the saccade was executed. By using the internal models, the salient objects are virtually fixated in both images. These fixated views are matched against each other using a simple difference-based matching approach. The performance of the model is evaluated through a large number of experiments on an image database and compared to a widely used approach from computer vision. In addition, a comparison on a commonplace scene is presented.

An efficient approach for robust multimodal retinal image registration based on UR-SIFT features and PIIFD descriptors

Existing algorithms based on scale invariant feature transform (SIFT) and Harris corners such as edge-driven dual-bootstrap iterative closest point and Harris-partial intensity invariant feature descriptor (PIIFD) respectivley have been shown to be robust in registering multimodal retinal images. However, they fail to register color retinal images with other modalities in the presence of large content or scale changes. Moreover, the approaches need preprocessing operations such as image resizing to do well. This restricts the application of image registration for further analysis such as change detection and image fusion. Motivated by the need for efficient registration of multimodal retinal image pairs, this paper introduces a novel integrated approach which exploits features of uniform robust scale invariant feature transform (UR-SIFT) and PIIFD. The approach is robust against low content contrast of color images and large content, appearance, and scale changes between color and other retinal image modalities like the fluorescein angiography. Due to low efficiency of standard SIFT detector for multimodal images, the UR-SIFT algorithm extracts high stable and distinctive features in the full distribution of location and scale in images. Then, feature points are adequate and repeatable. Moreover, the PIIFD descriptor is symmetric to contrast, which makes it suitable for robust multimodal image registration. After the UR-SIFT feature extraction and the PIIFD descriptor generation in images, an initial cross-matching process is performed and followed by a mismatch elimination algorithm. Our dataset consists of 120 pairs of multimodal retinal images. Experiment results show the outperformance of the UR-SIFT-PIIFD over the Harris-PIIFD and similar algorithms in terms of efficiency and positional accuracy.

Retinal image registration and comparison for clinical decision support.

For eye diseases, such as glaucoma and age-related macular degeneration (ARMD), involved in long-term degeneration procedure, longitudinal comparison of retinal images is a common step for reliable diagnosis of these kinds of diseases. To provide a retinal image registration approach for longitudinal retinal image alignment and comparison. Two image registration solutions were proposed for facing different image qualities of retinal images to make the registration methods more robust and feasible in a clinical application system. Thirty pairs of longitudinal retinal images were used for the registration test. The experiments showed both solutions provided good performance for the accurate image registrations with efficiency. We proposed a set of retinal image registration solutions for longitudinal retinal image observation and comparison targeting a clinical application environment.

Medial Axis for the Cortical Representation of 3D Shape

Representation of shape is a fundamental problem in vision science. Neurophysiological studies of macaque have reported that neurons in the primary visual cortex (V1) respond to Medial Axis (MA) of 2D surface, and that those in higher cortex (IT) respond to surfaces of 3D shape. However, how the cortex translates a pair of 2D retinal images into 3D shape has not been clarified. In the present study, we investigated the problem of 3D shape representation in the cortices with the specific focus on the translation of 2D to 3D shape representation. Based on the neurophysiological evidence, we propose that a pair of 2D-MA that encodes 2D surfaces in V1 is fused to generate a 3D-MA that represents 3D shape in higher cortex. In order to investigate this hypothesis, we developed a computational model and analyzed its behavior. The model computes 2D-MAs from a pair of stereo images, detects the disparities of the 2D-MAs using an energy model that mimics V1 complex cells, and computes 3D-MA from the disparities bet...