Fusion Of Image Features Research Articles

Visual Relationship Detection With Image Position and Feature Information Embedding and Fusion

Visual relationship detection (VRD) is an important direction in the field of image processing, and it is a research task to explore object relationships based on object recognition and localization regression. At the same time, it is also one of the key contents of scene graph generation and construction of multimodal knowledge graph. In order to better improve the visual relationship detection effect, the fusion method of image location and feature information embedding is adopted on the vrd dataset. First, analyze the causes of errors in existing methods, and build a pre-training pre-dataset vrd_P for data enhancement; then, combine the characteristics of the vrd dataset to establish an entities relationship network (Entities-net) to guide target recognition; finally, a visual relationship detection model of LTransE is proposed to achieve the joint representation of target features and target locations for relationship detection. The results show that the method of embedding fusion of image location and feature information can effectively improve the visual relationship detection effect.

Alzheimer's Disease Classification Based on Image Transformation and Features Fusion.

It has become an inevitable trend for medical personnel to analyze and diagnose Alzheimer's disease (AD) in different stages by combining functional magnetic resonance imaging (fMRI) and artificial intelligence technologies such as deep learning in the future. In this paper, a classification method was proposed for AD based on two different transformation images of fMRI and improved the 3DPCANet model and canonical correlation analysis (CCA). The main ideas include that, firstly, fMRI images were preprocessed, and subsequently, mean regional homogeneity (mReHo) and mean amplitude of low-frequency amplitude (mALFF) transformation were performed for the preprocessed images. Then, mReHo and mALFF images were extracted features using the improved 3DPCANet, and these two kinds of the extracted features were fused by CCA. Finally, the support vector machine (SVM) was used to classify AD patients with different stages. Experimental results showed that the proposed approach was robust and effective. Classification accuracy for significant memory concern (SMC) vs. mild cognitive impairment (MCI), normal control (NC) vs. AD, and NC vs. SMC, respectively, reached 95.00%, 92.00%, and 91.30%, which adequately proved the feasibility and effectiveness of the proposed method.

Gesture recognition using dual-stream CNN based on fusion of sEMG energy kernel phase portrait and IMU amplitude image

Since the surface electromyography signal (sEMG) is a weak and unstable signal, human action recognition based on sEMG time domain or frequency domain feature extraction has poor stability and low recognition rate. Based on modeling analysis of sEMG, the dual-stream convolutional neural network (CNN) is built to perform feature fusion of sEMG energy kernel phase portrait and inertial measurement unit(IMU) data for gesture recognition in this paper. Firstly, methods of matrix counting method and time window amplitude are utilized to represent the sEMG and IMU data as images. Secondly, the moving average method is presented to filter the images. Thirdly, the dual-stream CNN is used to perform feature extraction, fusion and classification of sEMG and IMU images respectively. Finally, the experiments yield an average recognition accuracy of 95.78% for 6 gestures performed by 5 subjects, which demonstrates that the proposed method has better performance for human gestures recognition.

Development and performance test of a vehicle-mounted total nitrogen content prediction system based on the fusion of near-infrared spectroscopy and image information

To measure the total nitrogen (TN) content of large-scale farmland soil accurately in real time and guide field fertilization, a vehicle-mounted soil TN content prediction system based on the fusion of near-infrared spectroscopy and image information was developed. A combined algorithm of uniform variable illumination (UVE) and adaptive weighted sampling (CARS) were used to select the characteristic wavelength, and the selected characteristic wavelength was used as the soil spectral information. Multiple linear regression model, partial least squares regression model, BP neural network prediction model and Catboost prediction model were chosen to predict soil TN content. After comparison, Catboost was finally selected as the final prediction model. The prediction system was composed of a mechanical part, an optical part, and a control part. The mechanical part provided space support for the prediction system, the optical part obtained the spectrum and image information of the soil, mainly including seven single-band near-infrared (NIR) filters: 945, 1045, 1200, 1300, 1450, 1535, and 1615 nm; 1450 nm is the sensitive wavelength which is used to eliminate the effect of water. The control part realized spectral data collection, image feature processing, data fusion, and the prediction of soil TN content. Soil samples were used to test the accuracy of the spectrum collection of the system. The test results showed that the highest correlation coefficient between the reflectance of the prediction system at seven sensitive wavelengths and the reflectance of the NIRQuest 512 infrared spectrometer is above 0.943. The field test results show that the R2 between the predicted soil TN content by the prediction system and the value of laboratory standard detection method is greater than 0.80 and the relative errors are less than 10%. The results also show that data fusion of spectral data and image features highlighted the common contribution of the near infrared spectrum and visible light images in the detection of soil TN content, and it can improve the prediction accuracy of the instrument. Moreover, the system can provide guidance for farmland scientific variable fertilization.

Research on Classification Model of Panax notoginseng Taproots Based on Machine Vision Feature Fusion.

The existing classification methods for Panax notoginseng taproots suffer from low accuracy, low efficiency, and poor stability. In this study, a classification model based on image feature fusion is established for Panax notoginseng taproots. The images of Panax notoginseng taproots collected in the experiment are preprocessed by Gaussian filtering, binarization, and morphological methods. Then, a total of 40 features are extracted, including size and shape features, HSV and RGB color features, and texture features. Through BP neural network, extreme learning machine (ELM), and support vector machine (SVM) models, the importance of color, texture, and fusion features for the classification of the main roots of Panax notoginseng is verified. Among the three models, the SVM model performs the best, achieving an accuracy of 92.037% on the prediction set. Next, iterative retaining information variables (IRIVs), variable iterative space shrinkage approach (VISSA), and stepwise regression analysis (SRA) are used to reduce the dimension of all the features. Finally, a traditional machine learning SVM model based on feature selection and a deep learning model based on semantic segmentation are established. With the model size of only 125 kb and the training time of 3.4 s, the IRIV-SVM model achieves an accuracy of 95.370% on the test set, so IRIV-SVM is selected as the main root classification model for Panax notoginseng. After being optimized by the gray wolf optimizer, the IRIV-GWO-SVM model achieves the highest classification accuracy of 98.704% on the test set. The study results of this paper provide a basis for developing online classification methods of Panax notoginseng with different grades in actual production.

Deepfake Detection Method Based on Cross-Domain Fusion

In recent years, despite its wide use in various fields, deepfake has been abused to generate hazardous contents such as fake movies, rumors, and fake news by manipulating or replacing facial information of the original sources and, thus, exerts huge security threats to the society. Facing the continuous evolution of deepfake, research on active detection and prevention technology becomes particularly important. In this paper, we propose a new deepfake detection method based on cross-domain fusion, which, on the basis of traditional spatial domain features, realizes the fusion of cross-domain image features by introducing edge geometric features of the frequency domain and, therefore, achieves considerable improvements on classification accuracy. Further evaluations of this method have been performed on publicly deepfake datasets, and the results show that our method is effective particularly on the Meso-4 DeepFake Database.

An Explainable Framework for Diagnosis of COVID-19 Pneumonia via Transfer Learning and Discriminant Correlation Analysis

The new coronavirus COVID-19 has been spreading all over the world in the last six months, and the death toll is still rising. The accurate diagnosis of COVID-19 is an emergent task as to stop the spreading of the virus. In this paper, we proposed to leverage image feature fusion for the diagnosis of COVID-19 in lung window computed tomography (CT). Initially, ResNet-18 and ResNet-50 were selected as the backbone deep networks to generate corresponding image representations from the CT images. Second, the representative information extracted from the two networks was fused by discriminant correlation analysis to obtain refined image features. Third, three randomized neural networks (RNNs): extreme learning machine, Schmidt neural network and random vector functional-link net, were trained using the refined features, and the predictions of the three RNNs were ensembled to get a more robust classification performance. Experiment results based on five-fold cross validation suggested that our method outperformed state-of-the-art algorithms in the diagnosis of COVID-19.

EFNet: evidence fusion network for tumor segmentation from PET-CT volumes

Precise delineation of target tumor from positron emission tomography-computed tomography (PET-CT) is a key step in clinical practice and radiation therapy. PET-CT co-segmentation actually uses the complementary information of two modalities to reduce the uncertainty of single-modal segmentation, so as to obtain more accurate segmentation results. At present, the PET-CT segmentation methods based on fully convolutional neural network (FCN) mainly adopt image fusion and feature fusion. The current fusion strategies do not consider the uncertainty of multi-modal segmentation and complex feature fusion consumes more computing resources, especially when dealing with 3D volumes. In this work, we analyze the PET-CT co-segmentation from the perspective of uncertainty, and propose evidence fusion network (EFNet). The network respectively outputs PET result and CT result containing uncertainty by proposed evidence loss, which are used as PET evidence and CT evidence. Then we use evidence fusion to reduce uncertainty of single-modal evidence. The final segmentation result is obtained based on evidence fusion of PET evidence and CT evidence. EFNet uses the basic 3D U-Net as backbone and only uses simple unidirectional feature fusion. In addition, EFNet can separately train and predict PET evidence and CT evidence, without the need for parallel training of two branch networks. We do experiments on the soft-tissue-sarcomas and lymphoma datasets. Compared with 3D U-Net, our proposed method improves the Dice by 8% and 5% respectively. Compared with the complex feature fusion method, our proposed method improves the Dice by 7% and 2% respectively. Our results show that in PET-CT segmentation methods based on FCN, by outputting uncertainty evidence and evidence fusion, the network can be simplified and the segmentation results can be improved.

Multi-scale spatial-spectral fusion based on multi-input fusion calculation and coordinate attention for hyperspectral image classification

Recently, the deep learning method that integrates image features has gradually become a hot development trend in hyperspectral image classification. However, these studies did not fully consider the fusion of image features, and did not remove the interference to the classification process caused by the difference in the size of the objects. These factors hinder the further improvement of the classification effect. To eliminate these drawbacks, this paper proposes a more effective fusion scheme (MSF-MIF), which realizes the fusion from the perspective of location characteristics and channel characteristics through 3D convolution and spatial feature concatenation. In view of the size discrepancy of the objects to be classified, this method extracts features from several input patches of different scales and uses the novel calculation method proposed to fuse them, which minimizes the interference caused by size differences. In addition, this research also tried to quote the coordinate attention structure for the first time that combines spatial and spectral attention features to further improve the classification performance. Experimental results on three commonly used data sets prove that this framework has achieved a breakthrough in classification accuracy.

Hyperspectral image classification with deep 3D capsule network and Markov random field

To address the existing problems of capsule networks in deep feature extraction and spatial-spectral feature fusion of hyperspectral images, this paper proposes a hyperspectral image classification method that combines a deep residual 3D capsule network and Markov random field. Based on this method, the deep spatial-spectral features of hyperspectral images are extracted using the deep residual 3D convolutional structure, the vector capsules of the features are obtained by the initial capsule layer and mapped into probability capsules via the 3D dynamic routing mechanism to construct the classification probability map, and the spatial structure of the classification results is regularised by the Markov random field to further improve the classification accuracy and performance of the images. Two sets of benchmark hyperspectral images, namely Indian Pines and Pavia University data sets, were used to conduct comparative experiments and ablation study. The experimental results showed that, compared with the conventional convolutional neural network and existing capsule network models, the proposed method not only improves the classification accuracy of the images but also partly eliminates the category noise and affords a more regular classification probability map.

Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information

Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.

An integrated multi-mode model of froth flotation cell based on fusion of flotation kinetics and froth image features

Froth flotation is the most commonly used technology in minerals separation and beneficiation. A comprehensive and precise flotation model is the foundation for optimizing the recovery rate and grade of valuable minerals. In this paper, a phenomenological model is built to describe the underlying physicochemical principles of the flotation process. To account for the influence factors not explicitly included in the phenomenological model, a data-driven model based on Long Short-Term Memory (LSTM) is proposed for error compensation by learning from the sequentially collected froth images. Considering the time-varying nature of flotation dynamics, a multi-mode modeling framework is established in the context of comprehensive state space (CSS) description system. In the framework, the dynamics of the flotation cell is described using a twofold integration. For each working condition, the phenomenological model and the compensation model are integrated to describe the process dynamics. The process dynamics is formulated as an integration of the sub-models of each working condition. In order to follow the evolution of working condition, the weights of the sub-models vary with the location of the working point in the CSS, which are calculated using a SAE(Stacked Auto Encoder)-KD(K-dimensional tree)-TAN(Tree Augmented Naive Bayes) approach. The proposed approach utilizes both kinetics knowledge and froth image features, and can be adapted to the variation of working conditions. A case study illustrates the stability and accuracy of the proposed approach.

Application of Two New Feature Fusion Networks to Improve Real-time Prostate Capsula Detection.

Excess prostate tissue is trimmed near the prostate capsula boundary during transurethral plasma kinetic enucleation of prostate (PKEP) and transurethral bipolar plasmakinetic resection of prostate (PKRP) surgeries. If a large portion of the tissue is removed, a prostate capsula perforation can potentially occur. As such, real-time accurate prostate capsula (PC) detection is critical for the prevention of these perforations. This study investigated the potential for using image denoising, image dimension reduction and feature fusion to improve real-time prostate capsula detection with two objectives. First, this paper mainly studied feature selection and input dimension reduction. Secondly, image denoising was evaluated, as it is of paramount importance to transient stability assessment based on neural networks. Two new feature fusion techniques, maxpooling bilinear interpolation single-shot multibox detector (PBSSD) and bilinear interpolation single shot multibox detector (BSSD) were proposed. Before original images were sent to the neural network, they were processed by principal component analysis (PCA) and adaptive median filter (AMF) for dimension reduction and image denoising. The results showed that the application of PCA and AMF with PBSSD increased the mean average precision (mAP) for prostate capsula images by 8.55% and reached 80.15%, compared with single shot multibox detector (SSD) alone. Application of PCA with BSSD increased the mAP for prostate capsula images by 4.6% compared with SSD alone. Compared with other methods, ours were proven to be more accurate for real-time prostate capsula detection. The improved mAP results suggest that the proposed approaches are powerful tools for improving SSD networks.

Water Extraction Method Based on Multi-Texture Feature Fusion of Synthetic Aperture Radar Images

Lakes play an important role in the water ecosystem on earth, and are vulnerable to climate change and human activities. Thus, the detection of water quality changes is of great significance for ecosystem assessment, disaster warning and water conservancy projects. In this paper, the dynamic changes of the Poyang Lake are monitored by Synthetic Aperture Radar (SAR). In order to extract water from SAR images to monitor water change, a water extraction algorithm composed of texture feature extraction, feature fusion and target segmentation was proposed. Firstly, the fractal dimension and lacunarity were calculated to construct the texture feature set of a water object. Then, an iterated function system (IFS) was constructed to fuse texture features into composite feature vectors. Finally, lake water was segmented by the multifractal spectrum method. Experimental results showed that the proposed algorithm accurately extracted water targets from SAR images of different regions and different imaging modes. Compared with common algorithms such as fuzzy C-means (FCM), the accuracy of the proposed algorithm is significantly improved, with an accuracy of over 98%. Moreover, the proposed algorithm can accurately segment complex coastlines with mountain shadow interference. In addition, the dynamic analysis of the changes of the water area of the Poyang Lake Basin was carried out with the local hydrological data. It showed that the extracted results of the algorithm in this paper are a good match with the hydrological data. This study provides an accurate monitoring method for lake water under complex backgrounds.

Image Retrieval Method Based on Image Feature Fusion and Discrete Cosine Transform

This paper presents a new content-based image retrieval (CBIR) method based on image feature fusion. The deep features are extracted from object-centric and place-centric deep networks. The discrete cosine transform (DCT) solves the strong correlation of deep features and reduces dimensions. The shallow features are extracted from a Quantized Uniform Local Binary Pattern (ULBP), hue-saturation-value (HSV) histogram, and dual-tree complex wavelet transform (DTCWT). Singular value decomposition (SVD) is applied to reduce the dimensions of ULBP and DTCWT features. The experimental results tested on Corel datasets and the Oxford building dataset show that the proposed method based on shallow features fusion can significantly improve performance compared to using a single type of shallow feature. The proposed method based on deep features fusion can slightly improve performance compared to using a single type of deep feature. This paper also tests variable factors that affect image retrieval performance, such as using principal component analysis (PCA) instead of DCT. The DCT can be used for dimensional feature reduction without losing too much performance.

Cross-modality image feature fusion diagnosis in breast cancer

Considering the complementarity of mammography and breast MRI, the research of feature fusion diagnosis based on cross-modality images was explored to improve the accuracy of breast cancer diagnosis. 201 patients with both mammography and breast MRI were collected retrospectively, including 117 cases of benign lesions and 84 cases of malignant ones. Two feature optimization strategies of sequential floating forward selection (SFFS), SFFS-1 and SFFS-2, were defined based on the sequential floating forward selection method. Each strategy was used to analyze the diagnostic performance of single-modality images and then to study the feature fusion diagnosis of cross-modality images. Three feature fusion approaches were compared: optimizing MRI features and then fusing those of mammography; optimizing mammography features and then fusing those of MRI; selecting the effective features from the whole feature set (mammography and MRI). Support vector machine, Naive Bayes, and K-nearest neighbor were employed as the classifiers and were finally integrated to get better performance. The average accuracy and area under the ROC curve (AUC) of MRI (88.56%, 0.9 for SFFS-1, 88.39%, 0.89 for SFFS-2) were better than mammography (84.25%, 0.84 for SFFS-1, 80.43%, 0.80 for SFFS-2). Furthermore, compared with a single modality, the average accuracy and AUC of cross-modality feature fusion can improve from 85.40% and 0.86 to 89.66% and 0.91. Classifier integration improved the accuracy and AUC from 90.49%, 0.92 to 92.37%, and 0.97. Cross-modality image feature fusion can achieve better diagnosis performance than a single modality. Feature selection strategy SFFS-1 has better efficiency than SFFS-2. Classifier integration can further improve diagnostic accuracy.

Approach on Recognizing Uncovered Earth Region from Aerial Images Based on Multi-feature Fusion

It is one of the main hidden dangers of power transmission lines accidents if there is the uncovered earth under or near power transmission lines. It can give the important early warning message to prevent the accidents through recognizing the uncovered earth region from aerial images of Unmanned Aerial Vehicle (UAV) power transmission lines inspection. Due to the low recognizing precision of Mask RCNN CNN (Mask Convolution Neural Network), this paper proposed an approach to recognize the uncovered earth region from aerial images of UAV by image feature fusion. The HOG and LBP features of aerial images were extracted and their dimension were reduced. Then these two features were fused by different weights. The experiments show that, (1) the average precision of recognizing the uncovered earth region can be above 80%, which is the lowest requirement to use; (2) the weights of two features should make the orders of magnitude of the two features as equal as possible. The approach is application for the first image filtering by the UAV airborne platform because it not only has enough good recognizing precision but also is very rapid, which provides a novel way for objective recognition from UAV aerial images.

CNN and multi-feature extraction based denoising of CT images

In the past decades, Computed Tomography (CT) images have been widely used and played a critically role in medical diagnosis. In low-dose CT images, reducing the radiation dose can reduce damage to patients, but at the same time, the projected image is contaminated with noise, resulting in a lot of noise in the reconstructed medical CT image, which can affect the clinical diagnosis. Based on the network structure of GoogleNet and Inception series, and combined with deep residual learning and convolutional neural networks, a novel denoising method with multi-feature extraction is proposed in this paper for medical CT images. The extraction of shallow multi-features in medical CT images is achieved by combining convolution filters of different sizes. This is useful for obtaining more detailed feature information in the image. Through the fusion of image features, the noise learning in medical CT images is realized. The developed neural network is more targeted to the removal of noise in the medical CT image. Experimental results show the denoised medical CT images can better retain edge and texture area details with the proposed method. Compared with the existing methods, the method proposed in this paper improves the denoising effect of medical CT images.

Recurrent feature fusion learning for multi-modality pet-ct tumor segmentation

Background and objective: [18f]-fluorodeoxyglucose (fdg) positron emission tomography – computed tomography (pet-ct) is now the preferred imaging modality for staging many cancers. Pet images characterize tumoral glucose metabolism while ct depicts the complementary anatomical localization of the tumor. Automatic tumor segmentation is an important step in image analysis in computer aided diagnosis systems. Recently, fully convolutional networks (fcns), with their ability to leverage annotated datasets and extract image feature representations, have become the state-of-the-art in tumor segmentation. There are limited fcn based methods that support multi-modality images and current methods have primarily focused on the fusion of multi-modality image features at various stages, i.e., early-fusion where the multi-modality image features are fused prior to fcn, late-fusion with the resultant features fused and hyper-fusion where multi-modality image features are fused across multiple image feature scales. Early- and late-fusion methods, however, have inherent, limited freedom to fuse complementary multi-modality image features. The hyper-fusion methods learn different image features across different image feature scales that can result in inaccurate segmentations, in particular, in situations where the tumors have heterogeneous textures. Methods: we propose a recurrent fusion network (rfn), which consists of multiple recurrent fusion phases to progressively fuse the complementary multi-modality image features with intermediary segmentation results derived at individual recurrent fusion phases: (1) the recurrent fusion phases iteratively learn the image features and then refine the subsequent segmentation results; and, (2) the intermediary segmentation results allows our method to focus on learning the multi-modality image features around these intermediary segmentation results, which minimize the risk of inconsistent feature learning. Results: we evaluated our method on two pathologically proven non-small cell lung cancer pet-ct datasets. We compared our method to the commonly used fusion methods (early-fusion, late-fusion and hyper-fusion) and the state-of-the-art pet-ct tumor segmentation methods on various network backbones (resnet, densenet and 3d-unet). Our results show that the rfn provides more accurate segmentation compared to the existing methods and is generalizable to different datasets. Conclusions: we show that learning through multiple recurrent fusion phases allows the iterative re-use of multi-modality image features that refines tumor segmentation results. We also identify that our rfn produces consistent segmentation results across different network architectures.

Joint embedding VQA model based on dynamic word vector.

The existing joint embedding Visual Question Answering models use different combinations of image characterization, text characterization and feature fusion method, but all the existing models use static word vectors for text characterization. However, in the real language environment, the same word may represent different meanings in different contexts, and may also be used as different grammatical components. These differences cannot be effectively expressed by static word vectors, so there may be semantic and grammatical deviations. In order to solve this problem, our article constructs a joint embedding model based on dynamic word vector—none KB-Specific network (N-KBSN) model which is different from commonly used Visual Question Answering models based on static word vectors. The N-KBSN model consists of three main parts: question text and image feature extraction module, self attention and guided attention module, feature fusion and classifier module. Among them, the key parts of N-KBSN model are: image characterization based on Faster R-CNN, text characterization based on ELMo and feature enhancement based on multi-head attention mechanism. The experimental results show that the N-KBSN constructed in our experiment is better than the other 2017—winner (glove) model and 2019—winner (glove) model. The introduction of dynamic word vector improves the accuracy of the overall results.