Multi-modal Image Sets Research Articles

An Efficient Algorithm for Multimodal Medical Image Fusion based On Feature Selection and PCA Using DTCWT

Background: In the two past decades, medical image fusion has become an essential part of modern medicine due to the availability of numerous imaging modalities (MRI, CT, SPECT. etc). This paper presents a new medical image fusion algorithm based on DTCWT and uses different fusion rules in order to obtain a new image which contains more information than any of the input images. Methods: In order to improve the visual quality of the fused image, we propose a new image fusion algorithm based on Dual Tree Complex Wavelet Transform (DTCWT). Using different fusion rules in a single algorithm leads to a perfect reconstruction of the output (fused image).This combination will create a new method which exploits the advantages of each method separately. DTCWT present good directionality since it considers the edge information in six directions and provides approximate shift invariant. The goal of Principal Component Analysis (PCA) is to extract the most significant features (wavelet coefficients in our case) in order to improve the spatial resolution. The proposed algorithm fuses the detailed wavelet coefficients of input images using features selection rule. Results: We have conducted several experiments over different sets of multimodal medical images such as CT/MRI, MRA/T1-MRI; however, only results of two sets have been presented (due to pages-limit). The proposed fusion algorithm is compared to recent fusion methods presented in the literature (eight methods) in terms of visual quality and quantitatively using well known fusion performance metrics (five metrics). Results showed that the proposed algorithm outperforms the existing ones in terms of visual and quantitative evaluations. Conclusion: This paper focuses on image fusion of medical images obtained from different modalities. We have proposed a novel algorithm based on DTCWT in order to merge multimodal medical images. Experiments have been performed over two different sets of multimodal medical images. The results show that the proposed method significantly outperforms other techniques reported in the literature.

Advanced Treatment Planning for Ophthalmic Plaque Brachytherapy of Uveal Melanomas

Brachytherapy (BT) treatments for uveal melanomas using ophthalmic plaques (i.e. 125I COMS and 106Ru eye plaques) achieves excellent clinical results with high tumor control and low toxicity rates. However, in comparison to state of the art external beam therapy and other BT techniques treatment planning of ophthalmologic plaque BT is underdeveloped regarding volumetric dose assessments, automated dose optimization routines and post-implantation quality assurance. The aim of this study was to (i) present a versatile and advanced treatment planning system (TPS) for ophthalmic plaque BT for both 125I COMS and 106Ru plaques and (ii) to demonstrate its potential in improving treatment planning. The TPS enables automated optimization of treatment plans based on Monte Carlo (MC) dose calculation and dose-volume histograms (DVH) as well as handling of multimodal image sets. In a first step, a resizable 3D eye model including all relevant critical structures was established, which enables to define both the tumor location and exact geometry by echography and funduscopy. Patient specific eye model adaptation can be achieved via post-implantation MR scans. An applicator library and respective dose look-up tables were generated by MC methods (MCNP6, Los Alamos, USA) that allow to generate DVHs for the target and all organs-at-risk (OAR). The current library of different plaques includes a 125I COMS plaque (Amersham Health model 6711) as well as several 106Ru plaques. The plaque geometry and 125I seeds were modeled according to AAPM TG 129 recommendations. The novel TPS allows calculating of full 3D dose distributions and DVH based on a cubic grid of 200 μm. The obtained results for the 125I COMS plaque and the values provided by AAPM TG 129 report were in good agreement. A maximum dose difference of 3 % up to a depth of 8 mm and 6 % up to a depth of 10 mm were obtained. Individualized modeling of the 3D eye based on fused funduscopic and post-implantation MR images allows to adjust treatment times and provide more accurate geometric definitions of the tumor and plaque location to improve the accuracy of the obtained DVHs. The implemented optimization routines can be used to define the optimal plaque size and safe eccentric plaque positions with respect to the tumor center to allow significant reductions of OAR doses. So far dose constrains for OARs (e.g. retinal doses of more than 500 Gy as a factor associated with visual acuity loss) were obtained from retrospective studies for treatment plan optimization. An advanced 3D TPS was developed and validated for plaque BT of uveal melanomas, which was designed to overcome the limitations of traditional treatment planning methods. The new TPS allows for a better and individualized treatment by including more image information and automated optimization into the planning process. The plaque library is currently updated to include more plaque types.

Simultaneous image fusion and denoising with adaptive sparse representation

In this study, a novel adaptive sparse representation (ASR) model is presented for simultaneous image fusion and denoising. As a powerful signal modelling technique, sparse representation (SR) has been successfully employed in many image processing applications such as denoising and fusion. In traditional SR-based applications, a highly redundant dictionary is always needed to satisfy signal reconstruction requirement since the structures vary significantly across different image patches. However, it may result in potential visual artefacts as well as high computational cost. In the proposed ASR model, instead of learning a single redundant dictionary, a set of more compact sub-dictionaries are learned from numerous high-quality image patches which have been pre-classified into several corresponding categories based on their gradient information. At the fusion and denoising processes, one of the sub-dictionaries is adaptively selected for a given set of source image patches. Experimental results on multi-focus and multi-modal image sets demonstrate that the ASR-based fusion method can outperform the conventional SR-based method in terms of both visual quality and objective assessment.

SEEG trajectory planning: combining stability, structure and scale in vessel extraction.

StereoEEG implantation is performed in patients with epilepsy to determine the site of the seizure onset zone. Intracranial haemorrhage is the most common complication associated to implantation carrying a risk that ranges from 0.6 to 2.7%, with significant associated morbidity. SEEG planning is done pre-operatively to identify avascular trajectories for the electrodes. In current practice neurosurgeons have no assistance in the planning of the electrode trajectories. There is great interest in developing computer assisted planning systems that can optimize the safety profile of electrode trajectories, maximizing the distance to critical brain structures. In this work, we address the problem of blood vessel extraction for SEEG trajectory planning. The proposed method exploits the availability of multi-modal images within a trajectory planning system to formulate a vessel extraction framework that combines the scale and the neighbouring structure of an object. We validated the proposed method in twelve multi-modal patient image sets. The mean Dice similarity coefficient (DSC) was 0.88 ± 0.03, representing a statistically significantly improvement when compared to the semi-automated single rater, single modality segmentation protocol used in current practice (DSC = 0.78 ± 0.02).

IMPROVING INDEPENDENT COMPONENT ANALYSIS USING SUPPORT VECTOR MACHINES FOR MULTIMODAL IMAGE FUSION

The objective of this study is to combine multiple images of a scene acquired by different sensors to create a new image with all important information from the i nput images. Recent studies show that bases trained using Independent Component Analysis (ICA) is effective in multisensor fusion and has improved performance over traditional wavelet approaches. In the ICA based fusion, the coefficients of the inpu t images are combined simply by selecting the coeffic ients with maximum magnitude. But this method resulted in fused images with poor contrast, due to the distortion introduced in constant background a reas. The performance of ICA based fusion can be greatly improved by using a region based approach with intelligent decision making in order to choose the significant regions in the source images. Hence, a new region based image fusion algorithm for combining visible and Infrared (IR) images using Independent Component Analysis and Support Vector Machines (SVM) is proposed. Region based joint segmentation of the source images is carried out in the spatial dom ain and important features of each region are compu ted in spatial and transform domain. A Support Vector Machine is trained to select the regions from the sourc e images with significant features and the correspond ing ICA coefficients are combined to form the fused ICA representation. The proposed algorithm is appli ed to different sets of multimodal images to valida te the robustness of the algorithm and compared with some standard image fusion methods. The fusion results demonstrate that the proposed scheme performs better than the state-of-the-art image fusion methods an d show a significant improvement in Entropy, Petrovic and Piella evaluation metrics.

Unification of image fusion and super-resolution using jointly trained dictionaries and local information contents

Multi-modal imaging requires image fusion to combine advantages of different types of sensors and requires super-resolution (SR) because of limited spatial resolution of source images. In this study, a novel framework is proposed for unification of image SR and the fusion process to obtain a high-resolution (HR)-fused image from a set of low-resolution (LR) multi-modal images. The jointly trained dictionaries of LR patches and corresponding HR patches are used for sparse representation of LR source image patches and HR-fused image patches, respectively. The sparse coefficients vectors for corresponding patches of source LR images are determined by using orthogonal matching pursuit and a local information content-based metric is employed to fuse these sparse coefficients. The corresponding HR-fused image patch is obtained by combining elements of the HR dictionary as per the fused coefficients of the LR image patches. The experimental results on sets of multi-modal images exhibited that the proposed method outperformed the existing fusion and SR techniques in terms of visual quality and image fusion quality metrics.

Fusion of multimodal medical images using Daubechies complex wavelet transform – A multiresolution approach

Multimodal medical image fusion is an important task for the retrieval of complementary information from medical images. Shift sensitivity, lack of phase information and poor directionality of real valued wavelet transforms motivated us to use complex wavelet transform for fusion. We have used Daubechies complex wavelet transform (DCxWT) for image fusion which is approximately shift invariant and provides phase information. In the present work, we have proposed a new multimodal medical image fusion using DCxWT at multiple levels which is based on multiresolution principle. The proposed method fuses the complex wavelet coefficients of source images using maximum selection rule. Experiments have been performed over three different sets of multimodal medical images. The proposed fusion method is visually and quantitatively compared with wavelet domain (Dual tree complex wavelet transform (DTCWT), Lifting wavelet transform (LWT), Multiwavelet transform (MWT), Stationary wavelet transform (SWT)) and spatial domain (Principal component analysis (PCA), linear and sharp) image fusion methods. The proposed method is further compared with Contourlet transform (CT) and Nonsubsampled contourlet transform (NSCT) based image fusion methods. For comparison of the proposed method, we have used five fusion metrics, namely entropy, edge strength, standard deviation, fusion factor and fusion symmetry. Comparison results prove that performance of the proposed fusion method is better than any of the above existing fusion methods. Robustness of the proposed method is tested against Gaussian, salt & pepper and speckle noise and the plots of fusion metrics for different noise cases established the superiority of the proposed fusion method.

Quantitative, Simultaneous PET/MRI for Intratumoral Imaging with an MRI-Compatible PET Scanner

Noninvasive methods are needed to explore the heterogeneous tumor microenvironment and its modulation by therapy. Hybrid PET/MRI systems are being developed for small-animal and clinical use. The advantage of these integrated systems depends on their ability to provide MR images that are spatially coincident with simultaneously acquired PET images, allowing combined functional MRI and PET studies of intratissue heterogeneity. Although much effort has been devoted to developing this new technology, the issue of quantitative and spatial fidelity of PET images from hybrid PET/MRI systems to the tissues imaged has received little attention. Here, we evaluated the ability of a first-generation, small-animal MRI-compatible PET scanner to accurately depict heterogeneous patterns of radiotracer uptake in tumors. Quantitative imaging characteristics of the MRI-compatible PET (PET/MRI) scanner were evaluated with phantoms using calibration coefficients derived from a mouse-sized linearity phantom. PET performance was compared with a commercial small-animal PET system and autoradiography in tumor-bearing mice. Pixel and structure-based similarity metrics were used to evaluate image concordance among modalities. Feasibility of simultaneous PET/MRI functional imaging of tumors was explored by following (64)Cu-labeled antibody uptake in relation to diffusion MRI using cooccurrence matrix analysis. The PET/MRI scanner showed stable and linear response. Activity concentration recovery values (measured and true activity concentration) calculated for 4-mm-diameter rods within linearity and uniform activity rod phantoms were near unity (0.97 ± 0.06 and 1.03 ± 0.03, respectively). Intratumoral uptake patterns for both (18)F-FDG and a (64)Cu-antibody acquired using the PET/MRI scanner and small-animal PET were highly correlated with autoradiography (r > 0.99) and with each other (r = 0.97 ± 0.01). On the basis of these data, we performed a preliminary study comparing diffusion MRI and radiolabeled antibody uptake patterns over time and visualized movement of antibodies from the vascular space into the tumor mass. The MRI-compatible PET scanner provided tumor images that were quantitatively accurate and spatially concordant with autoradiography and the small-animal PET examination. Cooccurrence matrix approaches enabled effective analysis of multimodal image sets. These observations confirm the ability of the current simultaneous PET/MRI system to provide accurate observations of intratumoral function and serve as a benchmark for future evaluations of hybrid instrumentation.

Sex, Aging, and Preexisting Cerebral Ischemic Disease in Patients With Aortic Stenosis

Patients undergoing cardiac surgery have a high frequency of preexisting cerebral ischemic lesions, the presence of which appears to predict cognitive sequelae. Patients undergoing aortic valve replacement for aortic stenosis (AS) incur an exceptionally high risk for perioperative cerebral ischemia. The extreme risk in this subgroup may arise from the preexisting burden of cerebral ischemic disease. We tested the hypotheses that increasing age, female sex, coronary artery disease, and the severity of AS are predictive of the severity of preexisting cerebral ischemic lesions. A total of 95 subjects were included in this study. Subjects were imaged on 1.5 Tesla magnetic resonance imaging scanners to obtain multimodal image sets which were used for the automatic segmentation of cerebral lesion volume. The dependence of lesion volume upon age, sex, coronary artery disease, and the severity of AS were tested. The results demonstrate a strong correlation between aging, female sex, and white matter and ischemia-like lesion volume in patients with aortic stenosis. Women and those of advanced age presenting for aortic valve replacement for AS may incur a particularly high risk for postoperative neurologic sequelae due to an exceptional preexisting burden of cerebral ischemic disease.

SU-FF-T-518: Implications of Radiation Treatment Planning Based On Apparent Diffusion Coefficient Map for Malignant Brain Tumors

Purpose: We investigated feasibility and advantage of integrating apparent diffusion coefficient (ADC) map for radiation palnning to determine the dose escalation region and evaluated the biolgocial planning reflected on tumor malignancy. Method and Materials: Multimodal image sets of glioma and astrocytoma were acquired with CT, contrast-enhanced T1, fluid attenuated inversion recovery (FLARE), and diffusion weighted (DW) magnetic resonance (MR) images. ADC map was obtained from DW images and it was converted into the ADC ratio map divided by the average ADC of normal parenchyma in contralateral brain. ADC ratio was used as quantitative criteria representing tumor malignancy and gross tumor volume (GTV) was differentiated by the level of the ADC ratio. GTV of each mulimodal images and stratified regions by ADC ratio were registered on CT using rigid registration. The biological planning using multmodal images was assessed by dose volume histogram and normal tissue complication probability (NTCP). Results: Multimodal images and ADC ratio map were complementary to describe more reliable tumor contour. More progreesed tumor region was distinguished in GTV and location of the most malignant tumor in 3D space was also resolved using the profile of ADC ratio. The advanced radiation treatment planning reflected on biological tumor characteristic was performed using the ADC map and dose escalation was achieved with similar NTCP level of conventional planning. Conclusion: ADC ratio was useful to stratify the tumor in regard to malignancy and it was suggeted a quantitative biological criteria to dermine the dose escalation region for effective treatment results. Biological treatment planning based on the ADC ratio map could support practical adaptive radiation therapy.

Joint Fusion and Blind Restoration For Multiple Image Scenarios With Missing Data

Image fusion systems aim at transferring ‘interesting’ information from the input sensor images to the fused image. The common assumption for most fusion approaches is the existence of a high-quality reference image signal for all image parts in all input sensor images. In the case that there are common degraded areas in at least one of the input images, the fusion algorithms cannot improve the information provided there, but simply convey a combination of this degraded information to the output. The authors propose a combined spatial-domain method of fusion and restoration in order to identify these common degraded areas in the fused image and use a regularized restoration approach to enhance the content in these areas. The proposed approach was tested on both multi-focus and multi-modal image sets and produced interesting results.

Multi-modal image set registration and atlas formation

In this paper, we present a Bayesian framework for both generating inter-subject large deformation transformations between two multi-modal image sets of the brain and for forming multi-class brain atlases. In this framework, the estimated transformations are generated using maximal information about the underlying neuroanatomy present in each of the different modalities. This modality independent registration framework is achieved by jointly estimating the posterior probabilities associated with the multi-modal image sets and the high-dimensional registration transformations mapping these posteriors. To maximally use the information present in all the modalities for registration, Kullback–Leibler divergence between the estimated posteriors is minimized. Registration results for image sets composed of multi-modal MR images of healthy adult human brains are presented. Atlas formation results are presented for a population of five infant human brains.

Automated registration of multimodal brain image sets using computer vision methods

We present a new method of registering three dimensional image volumes of the brain of a given patient acquired at different times or with different imaging modalities, or both. Registration is an essential requirement for fusing the data from the two image sets so as to either increase the available information by exploiting complementary imaging modalities, or to measure small changes over time for prognostication, disease assessment, etc. The new technique exploits an external, removable, remountable reference frame which is attached to the head. Computer vision techniques are used to determine the positions of fiducial marks in every image. The transformation required to map each image of one image volume onto the other image volume is developed using the theory of quaternions. The results indicate that the new technique is robust and practical in a clinical setting.