Image Fusion Process Research Articles

An efficient image segmentation and classification of lung lesions in pet and CT image fusion using DTWT incorporated SVM

Automatic segmentation and classification of fused lung Computed Tomography (CT) and Positron Emission Tomography (PET) images is presented. This system consists of four basic stages: 1). Lung image fusion process; 2). Segmentation of fused lung CT/PET images; 3). Post pre-processing; 4). Classification of fused lung images. In the first step, the lung image fusion process is made by deep learning method. At first, the input CT/PET images are decomposed by Dual Tree m-band Wavelet Transform (DTWT). The coefficients of DTWT are fused by deep learning method. This fused image of CT/PET is the input for the following steps. In the next step, the fused CT/PET images are decomposed by DTWT. It produces lower and higher frequency sub-band coefficients. Then the lower frequency components are set to zero. Then higher frequency components are used for reconstruction. Then the clustering-based thresholding method is used for segmentation. In post pre-processing step, the unwanted small regions are removed by morphological operations. Then the lung region is detected. At last, in the classification step, the features are extracted by the intensity and texture-based features. These features are classified by hybrid classifiers like Support Vector Machine (SVM) are used. The performance of the system has a higher classification accuracy of 99% using SVM classifier.

Image Fusion Based on NSST and CSR Under Robust Principal Component Analysis

Aiming at the problems of loss of detail information and noise interference that are easy to produce in the image fusion process, a robust principal component analysis (RPCA) based on Convolutional Sparse Coding (CSR) and For image fusion of NonSubsampled Shear Wave Transform (NSST), the source image is pre-enhanced first; then the image is decomposed by RPCA to obtain low-rank images and sparse images; then NSST fusion is used respectively For low-rank images, CSR coding is used to fuse sparse images, and finally two separately fused images are synthesized to obtain the final fused image. Experimental results show that the algorithm in this paper can effectively improve the contrast and clarity of the fused image, reduce noise interference, rich scene information, clear targets, and overall objective evaluation indicators are better than existing algorithms, and the operating efficiency has also been improved

Research on Risk Assessment of Multitarget Wireless Sensing Detection Auxiliary Engineering in Mine Geological Environment

In this paper, we evaluate the risk of auxiliary engineering of mine geological environment utilizing multiobjective wireless sensing detection. After vectorization of each evaluation factor, we evaluate the geological environment of a high‐top mountain mine by fuzzy comprehensive evaluation method and get the evaluation results of our geological environment. Firstly, an in‐depth analysis is carried out for the tracking gate that most association methods pay little attention to. In terms of optimization of traditional tracking gate, an attempt is made to propose a genetic algorithm fusion coding for tracking gate optimization, and then, the principal characteristics of traditional data association algorithm are focused on, and a particle swarm annealing association algorithm is proposed. Finally, the image information fusion of multisensors is studied, and the image information fusion level, fusion technology, and fusion process are discussed in detail, and the optimized discrete wavelet and color space alignment method is proposed, and good results are achieved through experiments. Among them, the area of poor geological environment evaluation is 0.32 km2, accounting for 6.6% of the whole study area, the area of average geological environment evaluation is 0.63 km2, accounting for 12.7% of the whole study area, and the area of good geological environment evaluation is 3.98 km2, accounting for 80.7% of the whole study area. The evaluation results can fully reflect the degree of mining influence on the Gaodingshan mining area and provide a theoretical basis for the study of mine environment restoration and management.

A neuro fuzzy image fusion using block based feature level method

Image fusion is a sub field of image processing in which more than one images are fused to create an image where all the objects are in focus. The process of image fusion is performed for multi-sensor and multi-focus images of the same scene. Multi-sensor images of the same scene are captured by different sensors whereas multi-focus images are captured by the same sensor. In multi-focus images, the objects in the scene which are closer to the camera are in focus and the farther objects get blurred. Contrary to it, when the farther objects are focused then closer objects get blurred in the image. To achieve an image where all the objects are in focus, the process of images fusion is performed either in spatial domain or in transformed domain. In recent times, the applications of image processing have grown immensely. Usually due to limited depth of field of optical lenses especially with greater focal length, it becomes impossible to obtain an image where all the objects are in focus. Thus it plays an important role to perform other tasks of image processing such as image segmentation, edge detection, stereo matching and image enhancement. Hence, a novel feature-level multi-focus image fusion technique has been proposed which fuses multi-focus images. Thus the results of extensive experimentation performed to highlight the efficiency and utility of the proposed technique is presented. The proposed work further explores comparison between fuzzy based image fusion and neuro fuzzy fusion technique along with quality evaluation indices.

A Contrast Adjusted Wavelet and Curvelet Transform Based Fusion of Breast Images

Restoration is one of the image processing techniques to restore the distorted image. The distorted image can be any type like blurred image, noise added either while storing or capturing of the images. the traditional restoration techniques are geometric correction, different types of filters to remove the noises. Generally, the restoration is applied to the single image to retrives its information. In this, the restoration process is applied in the application of image fusion of medical images. the reason to chose the image fusion process is to improve its entropy. The medical images used in this paper is the mammogram and MRI-breast image. In this, the image fusion is performed with the help of wavelet and curvelet transform. The curvelet transform is applied to the restored mammogram image with the help of contrast adjusted top-hat filter process. The wavelet transform is applied to the MRI image because it has more volume of data to acquire important details. The fused image is evaluated with the help of entropy, mean standard deviation and variance. The fused image has efficient value as compared to its original image but it is higher than the DTCWT-fused image. To verify the quality of the proposed method, the psnr is evaluated to determine the noise present in the image. From the evaluation, the noise is less and entropy is higher in the proposed method. Hence, the contrast adjusted wavelet curvelet fusion is the best restoration and fusion process in the breast images.

Small Multispectral UAV Sensor and Its Image Fusion Capability in Cultural Heritage Applications

For many decades the multispectral images of the earth’s surface and its objects were taken from multispectral sensors placed on satellites. In recent years, the technological evolution produced similar sensors (much smaller in size and weight) which can be placed on Unmanned Aerial Vehicles (UAVs), thereby allowing the collection of higher spatial resolution multispectral images. In this paper, Parrot’s small Multispectral (MS) camera Sequoia+ is used, and its images are evaluated at two archaeological sites, on the Byzantine wall (ground application) of Thessaloniki city (Greece) and on a mosaic floor (aerial application) at the archaeological site of Dion (Greece). The camera receives RGB and MS images simultaneously, a fact which does not allow image fusion to be performed, as in the standard utilization procedure of Panchromatic (PAN) and MS image of satellite passive systems. In this direction, that is, utilizing the image fusion processes of satellite PAN and MS images, this paper demonstrates that with proper digital processing the images (RGB and MS) of small MS cameras can lead to a fused image with a high spatial resolution, which retains a large percentage of the spectral information of the original MS image. The high percentage of spectral fidelity of the fused images makes it possible to perform high-precision digital measurements in archaeological sites such as the accurate digital separation of the objects, area measurements and retrieval of information not so visible with common RGB sensors via the MS and RGB data of small MS sensors.

Multimodal image fusion based on global-regional-local rule in NSST domain

An accurate representation of the source image is vital in image fusion processing. However, among most of the traditional multi-scale decomposition based image fusion methods, they fail to accurately interpret the source images, only capturing local information without considering the regional and global information. To overcome this drawback, a novel image fusion method based on the global-regional-local rule is proposed in the nonsubsampled shearlet transform (NSST) domain. First, the source paired images are decomposed into the low-pass subbands and high-pass subbands with the NSST transformation. Second, for comprehensively representing the statistical correlation of source image, three-level statistical models, that is, global contextual hidden Markov model (G-CHMM), regional contextual hidden Markov model (R-CHMM), and local contextual hidden Markov model (L-CHMM) are established for the high-pass subbands. In the R-CHMM, a novel feature vector is extracted for getting the region map of source image by the fuzzy cluster method (FCM). Third, to get accurate active measures of the source image, a global-regional-local fusion rule based on the statistical characteristics extracted from global-regional-local CHMM is designed, which is used to fuse the high-pass subbands. The low-pass subbands are fused with the choose-max rule based on the local gradient measure. Finally, the fused image is obtained by the inverse NSST. Experimental results on serials of infrared and visible images, medical images and multi-focus images demonstrate the advantage of the proposed method in terms of detail preserving.

Analysis and Identification of Tumor Cells In MRI And CT Images Based On Nanotechnology Using Neuro Fuzzy Image Fusion

Image fusion is a sub field of Medical imaging in which more than one images are fused to create an image where all the objects are in focus. The process of image fusion is performed for multi-sensor and multi-focus images of the same scene. Multi-sensor images of the same scene are captured by different sensors whereas multi-focus images are captured by the same sensor. In multi-focus images, the objects in the scene which are closer to the camera are in focus and the farther objects get blurred. Contrary to it, when the farther objects are focused then closer objects get blurred in the image. To achieve an image where all the objects are in focus, the process of images fusion is performed either in spatial domain or in transformed domain. In recent times, the applications of image processing have grown immensely. Usually due to limited depth of field of optical lenses especially with greater focal length, it becomes impossible to obtain an image where all the objects are in focus. Thus it plays an important role to perform tasks of analysis and identification of tumor cells in MRI and CT images based on neuro fuzzy image fusion. Hence, a novel feature-level multi-focus image fusion technique has been proposed which fuses multi-focus images. Thus the results of extensive experimentation performed to highlight the efficiency and utility of the proposed technique is presented. The proposed work further explores comparison between fuzzy based image fusion and neuro fuzzy fusion technique along with quality evaluation indices. In order to benefit from the highly conformal irradiation of, sophisticated treatment planning and simulation are required to analysis and identify the tumor in MRI and CT Scan images. The purpose of this study was to investigate the potential of MRI and CT for treatment plan simulation and adaptation using a neuro fuzzy image fusion.

Two-scale decomposition-based multifocus image fusion framework combined with image morphology and fuzzy set theory

Image fusion method can provide a high-quality image by merging the multiple features of different source images, and how to effectively evaluate the quality (informativeness) of image features is an important issue for image fusion. Because a considerable amount of imprecise and uncertain information exists in image fusion processes, this paper proposes a framework based on fuzzy set theory to handle the vague features, and a set of hybrid optimization methods is also designed to improve the performance. First, the two-scale decomposition method is utilized to decompose the source images and obtain a set of corresponding subimages. Second, fuzzy set theory and local spatial frequency are employed to generate preliminary decision maps by evaluating the pixel quality of the subimages. Third, a morphological method and consistency verification are utilized to optimize the decision maps to extract the focused and unfocused regions. Finally, three schemes are designed to generate the fused images according to the optimized decision maps. The experimental results show that the proposed method can achieve competitive performance compared with other methods.

Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation

Multi-modal medical image fusion refers to the fusion of two or more medical images obtained by different imaging methods into one image. Multi-modal medical images contain a lot of useful information that helps doctors to make a diagnosis. In this study, a multi-modal medical image fusion method is proposed based on Laplacian pyramid (LP) decomposition and adaptive sparse representation (ASR). ASR was used to reduce the noise of high-frequency information in the image fusion process and it did not need a high redundancy dictionary as traditional sparse representation (SR) methods. The proposed fusion method first used the LP decomposition to split medical images into four images of different sizes. Then ASR was performed to fuse the decomposed four layers, respectively. Finally, the fused image was obtained by the inverse Laplace pyramid transform. Experimental results showed that the proposed method could effectively fuse the medical images with the detailed information perfectly integrated, and could also reduce the influences of artifacts, noise and block effect. The research results are of great significance in the field of medical image fusion.

Infrared and visible image fusion via joint convolutional sparse representation

Recently, convolutional sparse representation (CSR) has improved the preservation of details of source images in the fusion results. This is mainly because the CSR has a global representation character that can improve spatial consistency in image representation. However, during image fusion processing, since the CSR expresses infrared and visible images separately, it ignores connections and differences between them. Further, CSR-based image fusion is not able to retain both strong intensity and clear details in the fusion results. In this paper, a novel fusion approach based on joint CSR is proposed. Specifically, we establish a joint form based on the CSR. The joint form is able to guarantee spatial consistency during image representation while obtaining distinct features, such as visible scene details and infrared target intensity. Experimental results illustrate that our fusion framework outperforms traditional fusion frameworks of sparse representation.

A generative adversarial network with structural enhancement and spectral supplement for pan-sharpening

Pan-sharpening aims to obtain high-resolution multi-spectral images by fusing panchromatic images and low-resolution multi-spectral images though reasonable rules. This paper proposed a novel generative adversarial network for pan-sharpening, which utilizes the supplemented spectral information from low-resolution multi-spectral images and the enhanced structural information from panchromatic images to generate high-resolution multi-spectral images. Firstly, the forward differential operator is used to extract the spatial structural information of the panchromatic image both in the horizontal and vertical directions. Secondly, an architecture of generative adversarial network is designed. The enhanced structural information generated by the accumulation of the structural information of the two directions is added to the image fusion process in generator and the discriminating process in discriminator, and a new optimization objective is designed accordingly. What is more, the low-resolution multi-spectral image is added to the convolution process in the generator as a supplement to the spectral information. Finally, in order to obtain better image generation effect, a special objective function of the generator is designed, which adds a unique relationship to reduce the loss of spatial structural information and spectral information of fused images. Experiments on QuickBird and WorldView-3 satellites datasets show that the proposed method can generate high quality fused images and is better than most advanced methods in both objective indicators and intuitive observations.

Recognition of Fusion Processing Based on Infrared and Visible Images under the Framework of Semi-supervised Learning

In traditional convolutional networks, due to the lack of adequate information protection of traditional image fusion technology and the incomplete removal of redundant noise, the useful information of the fusion image is missing and the recognition success rate is low. In this paper, through the research of deep learning-based image fusion methods and traditional target recognition and SVM neural network, an image fusion processing recognition method based on infrared and visible light is designed. The coding network of this image fusion method consists of convolutional layers, fusion layers and dense blocks. The output of each layer needs to be connected to the next few layers by using a densely connected neural network, so as to obtain more useful features from the source image and fuse the data of the two images better. It is verified by simulation that the fused image has sound visual effects, and its edges and details have been completely preserved. Thus, the target object has strong recognizability compared with the surrounding environment. Research shows that this method will help to more accurately interpret target information in complex environments and achieve more effective results in target recognition.

Infrared Image Enhancement by Multi-Modal Sensor Fusion in Enhanced Synthetic Vision System

ESVS (Enhanced Synthetic Vision System) employs multi-modal sensor fusion to provide equivalent out-of-cabin view for pilots. By using such device, pilots can identify the runway and obstacles even in low visibility weather condition during approach and landing. Previously, short-wave infrared video was used as real-time image sensor in our developed ESVS. While, compared to visible image, infrared image is not easy to understand. Moreover, the image quality of infrared degrades to a considerable extent in haze weather condition. To improve the infrared image visibility in ESVS, we proposed a multi-modal sensor fusion strategy by combining real-time infrared video and previous recorded visible video in the clear weather condition. During image fusion, considerable color detail of the visible image is injected to infrared image to ameliorate the infrared image quality. Furthermore, the proposed fusion method has two extra advantages. First, the important tagged information including runway and obstacles in visible image are transferred to infrared video, and second, the missing information in infrared vision can be complemented. To evaluate our proposed method, we use the Y-12F aircraft equipped with both visible and infrared video camera for flight test data collection. By collecting visible and infrared video in clear and haze weather conditions, as well as necessary navigation data, we carry out the final image fusion processing. Experimental results show that the fused video frame has enhanced image quality and improved readability for the pilots, which will significantly promote the pilots’ situation awareness during approach and landing. Thus, the proposed idea for fusion of multi-sensor image has the great potential to be applied to other cockpit display systems.

The refined spatiotemporal representation of soil organic matter based on remote images fusion of Sentinel-2 and Sentinel-3

Remote sensing technology is important for soil organic matter (SOM) estimation, but existing studies have mainly relied on a single data source. This limitation makes it difficult to simultaneously ensure high spatial resolution, high spectral accuracy and refined temporal granularity simultaneously, which cannot meet the requirements of the spatiotemporal dynamics representation. This study aimed to introduce a new remote sensing image source into SOM modeling and spatiotemporal estimation generated by fusing together Sentinel-2 and Sentinel-3 remote sensing images that have a 5-day revisit cycle; 10 m spatial resolution; and 21 different bands in blue, green, red and NIR spectral ranges. According to the image fusion process, a total of 52 available images were acquired between November 2016 and December 2018 in Donghai County, China. The fused images were used for SOM estimation model associated with 107 field samples. The results indicated that, first, the optimal model consisted of the band reflectivity (B20) and RVI (B18/B9), which were derived from the fused images, and the R2 approached 0.7 in the two phases of the synchronized data. Second, the modeling accuracy was influenced to some extent by the actual SOM content. The R2 values exceeded 0.75 when the SOM content was higher than 24 g/kg, while the R2 was even lower than 0.35 when the SOM content was lower. Third, the averaged SOM contents remained stable in general, while the seasonal variances can also be found during the two-year interval. The SOM contents maintained a low level during autumn and winter, while higher SOM levels were found in the spring and summer. Finally, the spatial variations could be described as ‘low in the west and high in the east’. In summary, the spatiotemporal dynamics of SOM highlighted the necessity of modeling with fused remote sensing images, and more effective modeling could be expected with the continued increase in SOM in future.

Adaptive image enhancement method using contrast limitation based on multiple layers BOHE

Multiple layers block overlapped histogram equalization (MLBOHE) is a classic image enhancement method. However, median filter is used in it to reduce noises which cause the degeneration of the local information. Moreover, the hidden details are not revealed effectively during the image fusion processes. To solve these drawbacks, an adaptive image enhancement method using contrast limitation is proposed in this paper. Based on MLBOHE, the proposed method employs a contrast limited method to suppress noises before BOHE is performed in each layer sub-blocks. Then an improved image fusion mode is applied to adaptively merge the multilayered BOHE images. The way to obtain the fusion weights by this fusion mode is according to the entropy value of each layer sub-blocks. In addition, four Image Quality Measures (IQMs), namely peak signal-to-noise ratio (PSNR), image clarity, contrast measure (EME) and feature similarity index metric (FSIM), are used to analyze the effectiveness of the proposed method. Simulation results show that the proposed method has high performance in suppressing noises and displaying more trustworthy details. Besides, this method outperforms the existing methods in weakening the excessive enhancement for low illumination and foggy images.

Image fusion using hybrid methods in multimodality medical images.

An image fusion based on multimodal medical images renders a considerable enhancement in the quality of fused images. An effective image fusion technique produces output images by preserving all the viable and prominent information gathered from the source images without any introduction of flaws or unnecessary distortions. This review paper intends to bring out the process of image fusion, its utilization in the medical domain, merits, and demerits and reviews the perspective of multimodal medical image fusion. It also discusses the involvement of various medical entities like medical resonance imaging (MRI), positron emission tomography (PET), and computed tomography (CT). The usefulness of such modalities is presented, suggesting plausible hybrid modality combinations which could greatly enhance image fusion. This review also discusses innovative dispositions in the medical image fusion techniques for the achievement of incisively desired, quality images focused on fusion with wavelet transform and use of independent component analysis (ICA) and principal component analysis (PCA) techniques for the purpose denoising and data dimension reductions. Additionally, the future-prospects of an ideal technique for medical image fusion through the utilization of various medical modalities have been also discussed in this review paper. Graphical abstract.

Fusion of brain MRI and CT images and Classification of brain tumor using Machine Learning Algorithm

Brain tumor is the most common and destructive disease which reduces the life time of people. The earlier detection of brain tumor plays a most important role for better treatment of the patient. In this paper, a new technique for brain tumor classification using machine learning by fusion of MRI and CT images are proposed. Image fusion is a process of fusing two or more images (i.e. MRI and CT scan images) to obtain a new one which contains more accurate information of the brain than any of the individual source images. Initially fusion of MRI and CT scan images has been carried out using Stationary Wavelet Transform (SWT). Then watershed transform is applied for image segmentation and discriminative robust local binary patter (DRLBP)is employed to extract the features exactly from the fused image. Classification of the tumor is done by Support Vector Machine (SVM) thereby reducing the generalization error and increasing the accuracy. The ultimate goal is to classify the tissues into normal and abnormal using machine learning algorithms .Image fusion process yields more accurate information of the brain than any of the individual source images.

Fusion of medical images using deep belief networks

Image fusion plays a significant role in various computer vision applications. However, designing an efficient image fusion technique is still a challenging task. In this paper, a novel deep belief networks based image fusion framework is proposed to improve the performance of the image fusion process further. We have initially, evaluated the fusion dataset by applying various feature extraction techniques. Thereafter, features selection techniques are applied to select potential features. Finally, the image fusion machine learning model is built by using a deep belief network model. Extensive experiments reveal that the proposed technique outperforms existing image fusion techniques.

Medical image fusion based on modified pulse coupled neural network model and kirsch operator

The medical image fusion process integrates the information of multiple source images into a single image. This fused image can provide more comprehensive information and is helpful in clinical diagnosis and treatment. In this paper, a new medical image fusion algorithm is proposed. Firstly, the original image is decomposed into a low-frequency sub-band and a series of high-frequency sub-bands by using nonsubsampled shearlet transform (NSST). For the low-frequency sub-band, kirsch operator is used to extract the directional feature maps from eight directions and novel sum-modified-Laplacian (NSML) method is used to calculate the significant information of each directional feature map, and then, combining a sigmod function and the significant information updated by gradient domain guided image filtering (GDGF), calculate the fusion weight coefficients of the directional feature maps. The fused feature map is obtained by summing the convolutions of the weight coefficients and the directional feature maps. The final fused low-frequency sub-band is obtained by the linear combination of the eight fused directional feature maps. The modified pulse coupled neural network (MPCNN) model is used to calculate the firing times of each high-frequency sub-band coefficient, and the fused high-frequency sub-bands are selected according to the firing times. Finally, the inverse NSST acts on the fused low-frequency sub-band and the fused high-frequency sub-bands to obtain the fused image. The experimental results show that the proposed medical image fusion algorithm expresses some advantages over the classical medical image fusion algorithms in objective and subjective evaluation.