Feature-level Image Fusion Research Articles

A Deep Feature Fusion Method for Complex Ground Object Classification in the Land Cover Ecosystem Using ZY1-02D and Sentinel-1A

Despite the successful application of multimodal deep learning (MDL) methods for land use/land cover (LULC) classification tasks, their fusion capacity has not yet been substantially examined for hyperspectral and synthetic aperture radar (SAR) data. Hyperspectral and SAR data have recently been widely used in land cover classification. However, the speckle noise of SAR and the heterogeneity with the imaging mechanism of hyperspectral data have hindered the application of MDL methods for integrating hyperspectral and SAR data. Accordingly, we proposed a deep feature fusion method called Refine-EndNet that combines a dynamic filter network (DFN), an attention mechanism (AM), and an encoder–decoder framework (EndNet). The proposed method is specifically designed for hyperspectral and SAR data and adopts an intra-group and inter-group feature fusion strategy. In intra-group feature fusion, the spectral information of hyperspectral data is integrated by fully connected neural networks in the feature dimension. The fusion filter generation network (FFGN) suppresses the presence of speckle noise and the influence of heterogeneity between multimodal data. In inter-group feature fusion, the fusion weight generation network (FWGN) further optimizes complementary information and improves fusion capacity. Experimental results from ZY1-02D satellite hyperspectral data and Sentinel-1A dual-polarimetric SAR data illustrate that the proposed method outperforms the conventional feature-level image fusion (FLIF) and MDL methods, such as S2ENet, FusAtNet, and EndNets, both visually and numerically. We first attempt to investigate the potentials of ZY1-02D satellite hyperspectral data affected by thick clouds, combined with SAR data for complex ground object classification in the land cover ecosystem.

Deep Learning Model for the Image Fusion and Accurate Classification of Remote Sensing Images.

Deep learning is widely used for the classification of images that have various attributes. Image data are used to extract colour, texture, form, and local features. These features are combined in feature-level image fusion to create a merged remote sensing image. A trained depth belief network (DBN) processes and divides fusion images, while a Softmax classifier determines the land type. As tested, the proposed approach can categorise all types of land. Traditional methods of detecting distant sensing photographs have limitations that can be overcome by using convolutional neural networks (CNN). Traditional techniques are incapable of combining deep learning elements while doing badly in classification. After PCA decreases data dimensionality, deep learning is applied to generate effective features that employ deep learning after PCA has reduced the dimensionality of the data. Principal component analysis is commonly used because of its effectiveness in attaining linear dimension reduction. It may be used on its own or as a starting point for further study into various different dimensionality reduction approaches. Data can be altered by remapping onto a new set of orthogonal axes using a process known as projection-based principal component analysis. Following remote sensing of land resources, the pictures were classified using a support vector machine. Euroset satellite images are used to assess the suggested approach. Accuracy and kappa have both increased. It was accurate and within 95.83 % of the planned figures. The classification findings' kappa value and reasoning time were 95.87 % and 128 milliseconds, respectively. Both the model's performance and the classification effect are excellent.

Brain tumor segmentation in multimodal MRI via pixel-level and feature-level image fusion.

Brain tumor segmentation in multimodal MRI volumes is of great significance to disease diagnosis, treatment planning, survival prediction and other relevant tasks. However, most existing brain tumor segmentation methods fail to make sufficient use of multimodal information. The most common way is to simply stack the original multimodal images or their low-level features as the model input, and many methods treat each modality data with equal importance to a given segmentation target. In this paper, we introduce multimodal image fusion technique including both pixel-level fusion and feature-level fusion for brain tumor segmentation, aiming to achieve more sufficient and finer utilization of multimodal information. At the pixel level, we present a convolutional network named PIF-Net for 3D MR image fusion to enrich the input modalities of the segmentation model. The fused modalities can strengthen the association among different types of pathological information captured by multiple source modalities, leading to a modality enhancement effect. At the feature level, we design an attention-based modality selection feature fusion (MSFF) module for multimodal feature refinement to address the difference among multiple modalities for a given segmentation target. A two-stage brain tumor segmentation framework is accordingly proposed based on the above components and the popular V-Net model. Experiments are conducted on the BraTS 2019 and BraTS 2020 benchmarks. The results demonstrate that the proposed components on both pixel-level and feature-level fusion can effectively improve the segmentation accuracy of brain tumors.

MsIFT: Multi-Source Image Fusion Transformer

Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.

A feature level image fusion for Night-Vision context enhancement using Arithmetic optimization algorithm based image segmentation

Images are fused to produce a composite image by combining key characteristics of the source images in image fusion. It makes the fused image better for human vision and machine vision. A novel procedure of Infrared (IR) and Visible (Vis) image fusion is proposed in this manuscript. The main challenges of feature level image fusion are that it will introduce artifacts and noise in the fused image. To preserve the meaningful information without adding artifacts from the source input images, weight map computed from Arithmetic optimization algorithm (AOA) is used for the image fusion process. In this manuscript, feature level fusion is performed after refining the weight maps using a weighted least square optimization (WLS) technique. Through this, the derived salient object details are merged into the visual image without introducing distortion. To affirm the validity of the proposed methodology simulation results are carried for twenty-one image data sets. It is concluded from the qualitative and quantitative experimental analysis that the proposed method works well for most of the image data sets and shows better performance than certain traditional existing models.

A feature level image fusion for IR and visible image using mNMRA based segmentation

A feature level image fusion for IR and visible image using mNMRA based segmentation

Remote Sensing Image Land Classification Based on Deep Learning

Aiming at the problems of high-resolution remote sensing images with many features and low classification accuracy using a single feature description, a remote sensing image land classification model based on deep learning from the perspective of ecological resource utilization is proposed. Firstly, the remote sensing image obtained by Gaofen-1 satellite is preprocessed, including multispectral data and panchromatic data. Then, the color, texture, shape, and local features are extracted from the image data, and the feature-level image fusion method is used to associate these features to realize the fusion of remote sensing image features. Finally, the fused image features are input into the trained depth belief network (DBN) for processing, and the land type is obtained by the Softmax classifier. Based on the Keras and TensorFlow platform, the experimental analysis of the proposed model shows that it can clearly classify all land types, and the overall accuracy, F1 value, and reasoning time of the classification results are 97.86%, 87.25%, and 128 ms, respectively, which are better than other comparative models.

Multispectral Palm Print and Palm Vein Acquisition Platform and Recognition Method Based on Convolutional Neural Network

Abstract Biometrics recognition takes advantage of feature extraction and pattern recognition to analyze the physical and behavioral characteristics of biological individuals to achieve the purpose of individual identification. As a typical biometric technology, palm print and palm vein have the characteristics of high recognition rate, stable features, easy location and good image quality, which have attracted the attention of researchers. This paper designs and develops a multispectral palm print and palm vein acquisition platform, which can quickly acquire palm spectrum and palm vein multispectral images with seven different wavelengths. We propose a multispectral palm print palmar vein recognition framework, and feature-level image fusion is performed after extracting features of palm print palmar vein images at different wavelengths. Through the multispectral palm print palm vein image fusion experiment, a more feasible multispectral palm print and palm vein image fusion scheme is proposed. Based on the results of image fusion, we further propose an improved convolutional neural network (CNN) for model training to achieve identity recognition based on multispectral palm print palm vein images. Finally, the effects of different CNN network structures and learning rates on the recognition results were analyzed and compared experimentally.

Feature-Level Fusion between Gaofen-5 and Sentinel-1A Data for Tea Plantation Mapping

The accurate mapping of tea plantations is significant for government decision-making and environmental protection of tea-producing regions. Hyperspectral and Synthetic Aperture Radar (SAR) data have recently been widely used in land cover classification, but effective integration of these data for tea plantation mapping requires further study. This study developed a new feature-level image fusion method called LPPSubFus that combines locality preserving projection and subspace fusion (SubFus) to map tea plantations. Based on hyperspectral and SAR data, we first extracted spectral indexes, textures, and backscattering information. Second, this study applied LPPSubFus to tea plantation mapping with different classification algorithms. Finally, we compared the performance of LPPSubFus, SubFus, and pixel-level image fusion in tea plantation mapping. Feature-level image fusion performed better than pixel-level image fusion. An improvement of about 3% was achieved using feature-level image fusion compared to hyperspectral data alone. Regarding feature-level image fusion, LPPSubFus improved the overall accuracy by more than 3% compared to SubFus. In particular, LPPSubFus using neural network algorithms achieved the highest overall accuracy (95%) and over 90% producer and user accuracy for tea plantations and forests. In addition, LPPSubFus was more compatible with different classification algorithms than SubFus. Based on these findings, it is concluded that LPPSubFus has better and more stable performance in tea plantation mapping than pixel-level image fusion and SubFus. This study demonstrates the potential of integrating hyperspectral and SAR data via LPPSubFus for mapping tea plantations. Our work offers a promising tea plantation mapping method and contributes to the understanding of hyperspectral and SAR data fusion.

Multi-Modal Image Fusion via Convolutional Morphological Component Analysis and Guided Filter

In feature-level image fusion, deep learning technology, particularly convolutional sparse representation (SR) theory, has emerged as a new topic over the past three years. This paper proposes an effective image fusion method based on convolution SR, namely, convolutional sparsity-based morphological component analysis and guided filter (CS-MCA-GF). The guided filter operator and choose-max coefficient fusion scheme introduced in this method can effectively eliminate the artifacts generated by the morphological components in the linear fusion, and maintain the pixel saliency of the source images. Experiments show that the proposed method can achieve an excellent performance in multi-modal image fusion, which includes medical image fusion.

Application of Pseudo-color Image Feature-Level Fusion in Nondestructive Testing of Wire Ropes

Among the many nondestructive testing methods of ferromagnetic materials, magnetic flux leakage (MFL) testing is a very mature technology. Due to the single source of target information in traditional MFL detection technology, the recognition accuracy is low. Infrared image is a color artificially given, so it is also a pseudo-color image. We use the feature-level image fusion technology to fuse color infrared images and pseudo-color MFL images and quantitatively identify the defect feature vectors after fusion. A wavelet noise reduction algorithm is used to perform noise reduction processing on the MFL signal and used the pseudo-color transform method to convert the noise-reduced MFL signal into a pseudo-color image. For the collected infrared images of the wire rope, we use an image segmentation algorithm to extract the defective parts. The color moments and texture features of the pseudo-color MFL defect image and the color infrared defect image are extracted for feature-level fusion and trained as the input of the nearest neighbor algorithm. The final experimental results prove that the fusion of the two images information has achieved good results and the defect recognition rate has been improved.

A Methodical Block based Feature Level Image Fusion Technique with Wavelet Transform using Neural Network for Satellite Images

This paper describe about the feature extraction or detection machine learning application which one is wavelet transform integrated with neural network. It has obtained an effective block based feature level with wavelet transform using neural network (BFWN) model for image fusion. In the projected BFWN model, the discrete wavelet transform (DWT) and neural network (NN) are considered for fusing IRS-1D images using LISS- III scanner about the location different areas in India. Also Quick Bird image data and Landsat 7 image data are used to carry out on the proposed BFWN method. The characteristics like contrast visibility, energy of gradient, spatial frequency, variance and edge information are under study. A Feed forward back propagation neural network is trained and tested for categorization since the learning capability of neural network makes it feasible to customize the image fusion process. The trained neural network is used to fuse the two source images. The proposed BFWN model is distinguish, with DWT alone to assess the quality of the fused image. The results obviously show that the proposed BFWN model is a capable and feasible algorithm for image fusion

Feature level image fusion of optical imagery and Synthetic Aperture Radar (SAR) for invasive alien plant species detection and mapping

Invasive alien plant species are regarded as a major threat to among others socio-economic systems, global biodiversity and conservation initiatives. A reliable understanding of their spatial and temporal distribution is paramount for understanding their impact on co-existing landscapes and ecosystems. While traditional passive remote sensing methods have been successful in assessing invasion of such species, limiting factors such as cost, restricted coverage, image availability, terrain and inadequate resolutions hamper mapping and detection at large spatial extents. To date, the adoption of active remote sensing techniques as complimentary data to invasive alien plant mapping has been limited. In this study, we fuse two commonly used medium spatial and spectral resolution imagery (Sentinel-2 and Landsat 8) with active remote sensing data (Synthetic Aperture Rada imagery) in determining the optimal season for detecting and mapping the American Bramble (Rubus cuneifolius). Feature level image fusion was adopted to integrate passive and active remote sensing imagery and Support vector machine (SVM) supervised classification algorithm used to discriminate the American Bramble from surrounding native vegetation. Seasonal results showed that Sentinel-2 data, fused with SAR data generated the highest classification accuracy during summer (76%), while Landsat 8 imagery fused with SAR data performed best in winter (72%). These findings demonstrate that fusion of SAR with traditional optical imagery can be used to detect and map the American Bramble at a regional scale. We conclude that SAR data can be used synergistically with optical remote sensing to improve discrimination and mapping of the American Bramble.

Grey-Wolf Optimizer Based Feature Selection for Feature-Level Multi-Focus Image Fusion

This paper proposes optimal ensemble-individual-features (OEIF) for multi-focus image fusion through combining the decision information of individual features. This proposed system consists of three stages. In the first stage, the different types of features such as spatial, texture and frequency are extracted from every block on input blurred images. In the second step, grey wolf optimiser (GWO)-based features validation method is proposed to find suitable features from source images. This method is based on an iterative process, in which each individual represents a candidate solution for validating/invalidating the features. In the final step, the ensemble decision based on optimal individual features is utilised to fuse blurred images. We prove that OEIF method is better in comparison to the noisy feature-based individual pixel-level and the feature-level fusion methods with different multi-focus images and it reveals that OGWO-based proposed method performs better visual quality than other methods.

A Feature-level Image Fusion Performance Evaluating Measure based on Contourlet Transform

On the basis of the studying and summarizing the single-scale feature-level and multi-scale feature-level image fusion assessment method, we proposed a novel multi-scale, multi-directional image fusion performance evaluation measure. In this measure, we integrate the contourlet transform into the fusion evaluation, then, the similarity between source images and fused images are calculated in terms of edge and texture information. The better results are obtained after compared with the subjective evaluation results, as well as other evaluation measures. Experimental results show that the measure is reliable.

A Feature-level Image Fusion Performance Evaluating Measure based on Contourlet Transform

A Feature-level Image Fusion Performance Evaluating Measure based on Contourlet Transform

Improved Block Based Feature Level Image Fusion Technique Using Contourlet with Neural Network

Image fusion is defined as the process of combining two or more different images into a new single image retaining the important features from each image with extended information content. To overcome the spectral distortions in the fused image, many multiresolution based approaches have been proposed. They include different pyramid transforms and discrete scalar wavelet transforms. The multiwavelet transform (MWT) produces a non-redundant image representation, which provides better spatial and spectral localization of image formation. This paper has derived an efficient block based feature level multiwavelet transform with neural network (BFMN) model for image fusion. The proposed (BFMN) model integrates MWT with neural network, which is one of the feature extraction or detection machine learning applications. In the proposed BFMN model, the two fusion techniques, multiwavelet transform (MWT) and neural network (NN) are discussed for fusing the IRS-1D images using LISS III scanner about the locations Hyderabad, Vishakhapatnam, Mahaboobnagar and Patancheru in Andhra Pradesh, India. Also QuickBird image data and Landsat 7 image data are used to perform experiments on the proposed BFMN model. The features under study are contrast visibility, spatial frequency, energy of gradient, variance and edge information. Feed forward back propagation neural network is trained and tested for classification since the learning capability of neural network makes it feasible to customize the image fusion process. The trained neural network is then used to fuse the pair of source images. The proposed BFMN model is compared with other techniques to assess the quality of the fused image. Experimental results clearly prove that the proposed BFMN model is an efficient and feasible algorithm for image fusion.

An Efficient Block based Feature Level Image Fusion Technique using Wavelet Transform and Neural Network

fusion is a process of combining relevant information from two or more images into a single informative image. In this paper, wavelet transform is integrated with neural network, which is one of the feature extraction or detection machine learning applications. This paper has derived an efficient block based feature level wavelet transform with neural network (BFWN) model for image fusion. In the proposed BFWN model, the two fusion techniques, discrete wavelet transform (DWT) and neural network (NN) are discussed for fusing IRS-1D images using LISS III scanner about the location Hyderabad, Vishakhapatnam, Mahaboobnagar and Patancheru in India. Also QuickBird image data and Landsat 7 image data are used to perform experiments on the proposed BFWN method. The features under study are contrast visibility, spatial frequency, energy of gradient, variance and edge information. Feed forward back propagation neural network is trained and tested for classification since the learning capability of neural network makes it feasible to customize the image fusion process. The trained neural network is then used to fuse the pair of source images. The proposed BFWN model is compared with DWT alone to assess the quality of the fused image. Experimental results clearly prove that the proposed BFWN model is an efficient and feasible algorithm for image fusion.

Image fusion algorithm based on block-DCT in wavelet domain

A feature-level image fusion algorithm using wavelet transform combined with the discrete cosine transform (DCT) is proposed. The basic idea is to perform block-DCT of each source image first, and then to select the retained coefficients according to maximum variance; each image is compressed to 25 percent of the original size. The coordinates of the retained coefficients are used as private keys. Finally, the processed DCT coefficient matrixes are taken as the wavelet coefficients and the fused image is obtained by taking the inverse wavelet transform. The experimental results show that the algorithm realizes the fusion of images with different sizes. Furthermore, one single image can be reconstructed by one single private key.

A new multi-spectral feature level image fusion method for human interpretation

Various different methods to perform multi-spectral image fusion have been suggested, mostly on the pixel level. However, the jury is still out on the benefits of a fused image compared to its source images. We present here a new multi-spectral image fusion method, multi-spectral segmentation fusion (MSSF), which uses a feature level processing paradigm. To test our method, we compared human observer performance in a three-task experiment using MSSF against two established methods: averaging and principle components analysis (PCA), and against its two source bands, visible and infrared. The three tasks that we studied were: (1) simple target detection, (2) spatial orientation, and (3) camouflaged target detection. MSSF proved superior to the other fusion methods in all three tests; MSSF also outperformed the source images in the spatial orientation and camouflaged target detection tasks. Based on these findings, current speculation about the circumstances in which multi-spectral image fusion in general and specific fusion methods in particular would be superior to using the original image sources can be further addressed.