Deep Fusion Feature Research Articles

MAFNet: a two-stage multiple attention fusion network for partial-to-partial point cloud registration

Abstract 3D point cloud registration is a critical technology in the fields of visual measurement and robot automation processing. In actual large-scale industrial production, the accuracy of point cloud registration directly affects the quality of automated welding processes. However, most existing methods are confronted with serious challenges such as the failure of partial-to-partial point cloud model registration when facing robot automatic processing guidance and error analysis work. Therefore, this paper proposes a novel two-stage network architecture for point cloud registration, which aims at robot pose adjustment and visual guidance in the field of automated welding by using 3D point cloud data. Specifically, we propose a neighborhood-based multi-head attention module in the coarse registration stage. The neighborhood information of each point can be aggregated through focusing on different weight coefficients of multi-head inputs. Then the spatial structure features which is used to establish the overlapping constraint of point clouds are obtained based on above neighborhood information. In the fine registration stage, we propose the similarity matching removal module based on multiple attention fusion features to explore deeper features from different aspects. By using deep fusion features to guide the similarity calculation, the interference of non-overlapping points is removed to achieve the finer registration. Eventually, we compare and analyze the proposed method with the SOTA ones through several error metrics and overlap estimation experiments based on the ModelNet40 dataset. The test results indicate that our method, relative to other mainstream techniques, achieves lower error rates and the most superior accuracy of 98.61% and recall of 98.37%. To demonstrate the generalization performance of proposed algorithm, extensive experiments on the Stanford 3D Scanning Repository, 7-Scenes and our own scanning dataset using partially overlapping point clouds individually under clean and noisy conditions show the validity and reliability of our proposed registration network.

Adap-BDCM: Adaptive Bilinear Dynamic Cascade Model for Classification Tasks on CNV Datasets.

Copy number variation (CNV) is an essential genetic driving factor of cancer formation and progression, making intelligent classification based on CNV feasible. However, there are a few challenges in the current machine learning and deep learning methods, such as the design of base classifier combination schemes in ensemble methods and the selection of layers of neural networks, which often result in low accuracy. Therefore, an adaptive bilinear dynamic cascade model (Adap-BDCM) is developed to further enhance the accuracy and applicability of these methods for intelligent classification on CNV datasets. In this model, a feature selection module is introduced to mitigate the interference of redundant information, and a bilinear model based on the gated attention mechanism is proposed to extract more beneficial deep fusion features. Furthermore, an adaptive base classifier selection scheme is designed to overcome the difficulty of manually designing base classifier combinations and enhance the applicability of the model. Lastly, a novel feature fusion scheme with an attribute recall submoduleis constructed, effectively avoiding getting stuck in local solutions and missing some valuable information. Numerous experiments have demonstrated that our Adap-BDCM model exhibits optimal performance in cancer classification, stage prediction, and recurrence on CNV datasets. This study can assist physicians in making diagnoses faster and better.

Intelligent fault diagnosis of storage stacking machinery under variable working conditions using attention-based adaptive multimodal feature fusion networks

Due to the harsh working environment of storage stacking machinery, the fault information of important components is significantly complex, which leads to the problem of low classification accuracy and high computational complexity of existing deep learning-based fault diagnosis methods. To alleviate the problem, this paper presents a novel architecture named attention-based adaptive multimodal feature fusion networks for intelligent fault diagnosis of storage stacking machinery, which is aimed at improving the diagnostic precision and robustness of feature fusion network and learning the broader feature representation. Firstly, the long short-term memory layer is introduced to extract the feature information of multiple time steps to improve the self-extraction ability of multi-temporal features. Then, the maximum temporal feature fusion module is utilized to highlight the recognizability of deep fusion features. Finally, a residual layer with spanning connections is added to increase the utilization and characterization capability of deep fusion features. Experimental results demonstrate the effectiveness and superiority of the proposed method in intelligent fault diagnosis of storage stacking machinery under variable working conditions compared with some state-of-the-art deep learning-based methodologies.

Dilated Spectral–Spatial Gaussian Transformer Net for Hyperspectral Image Classification

In recent years, deep learning-based classification methods for hyperspectral images (HSIs) have gained widespread popularity in fields such as agriculture, environmental monitoring, and geological exploration. This is owing to their ability to automatically extract features and deliver outstanding performance. This study provides a new Dilated Spectral–Spatial Gaussian Transformer Net (DSSGT) model. The DSSGT model incorporates dilated convolutions as shallow feature extraction units, which allows for an expanded receptive field while maintaining computational efficiency. We integrated transformer architecture to effectively capture feature relationships and generate deep fusion features, thereby enhancing classification accuracy. We used consecutive dilated convolutional layers to extract joint low-level spectral–spatial features. We then introduced Gaussian Weighted Pixel Embedding blocks, which leverage Gaussian weight matrices to transform the joint features into pixel-level vectors. By combining the features of each pixel with its neighbouring pixels, we obtained pixel-level representations that are more expressive and context-aware. The transformed vector matrix was fed into the transformer encoder module, enabling the capture of global dependencies within the input data and generating higher-level fusion features with improved expressiveness and discriminability. We evaluated the proposed DSSGT model using five hyperspectral image datasets through comparative experiments. Our results demonstrate the superior performance of our approach compared to those of current state-of-the-art methods, providing compelling evidence of the DSSGT model’s effectiveness.

Intelligent fault diagnosis of rotating machinery under variable working conditions based on deep transfer learning with fusion of local and global time–frequency features

With the development of deep learning methods, the data-driven fault diagnosis methods have attracted a great deal of interest. However, as for the data-driven fault diagnosis methods, technology has to overcome various difficulties in the practical industrial scenarios, such as variable working conditions, insufficient effective samples, and environmental noise interference. Combining with the time–frequency analysis of vibration signals, a domain adaptation fault diagnosis model based on ResNet and Transformer (DAFDMRT) is proposed in this work, aiming to solve the problems encountered by current rotating machinery fault diagnosis methods in the field of application. Firstly, the vibration signal is processed by wavelet packet transform and the time–frequency information grayscale maps is constructed. Next, a deep fusion feature extraction network combining ResNet and Transformer encoder, is designed for the extraction and fusion of the local and global features of multi-scale time–frequency information. Finally, the multi-kernel maximum mean discrepancy is applied to measure and minimize the distribution difference between the deep features of source and target domain, thereby improving the diagnostic performance of the diagnosis model in variable working conditions. In this work, comparative experiments are conducted as for bearing and gearbox datasets under variable working conditions. The results indicate that DAFDMRT can show excellent performances in terms of fault diagnosis and generalization ability.

Deep sample clustering domain adaptation for breast histopathology image classification

Deep learning has been widely applied to the diagnosis of breast cancer. Most deep learning models require a large number of labeled samples for training, but a large number of datasets with detailed labels is impractical. Therefore, exploring the methods to accurately classify unlabeled data remains a crucial area of research. In this paper, a deep sample clustering unsupervised domain adaptation method based on deep fusion feature is proposed for breast histopathology image classification. Deep multi-level fusion feature has a strong ability to capture image information, allowing for accurate characterization of the structural features of cells, thus improving the clustering performance. Next, based on the criterion of adaptive clustering of samples, the similarity between different clustering centers in the source and target domains is adjusted jointly. Finally, the classification results were achieved by the nearest neighbor relationship between the clustering centers and samples in the target domain. The proposed method was tested on three different breast cancer datasets. The prediction results of all patches were aggregated to obtain the classification accuracy of the histopathological image. The experimental results demonstrate that the proposed method achieves high classification accuracy and has the potential for application in clinical settings.

Artificial intelligence method based on multi-feature fusion for automatic macular edema (ME) classification on spectral-domain optical coherence tomography (SD-OCT) images.

A common ocular manifestation, macular edema (ME) is the primary cause of visual deterioration. In this study, an artificial intelligence method based on multi-feature fusion was introduced to enable automatic ME classification on spectral-domain optical coherence tomography (SD-OCT) images, to provide a convenient method of clinical diagnosis. First, 1,213 two-dimensional (2D) cross-sectional OCT images of ME were collected from the Jiangxi Provincial People's Hospital between 2016 and 2021. According to OCT reports of senior ophthalmologists, there were 300 images with diabetic (DME), 303 images with age-related macular degeneration (AMD), 304 images with retinal-vein occlusion (RVO), and 306 images with central serous chorioretinopathy (CSC). Then, traditional omics features of the images were extracted based on the first-order statistics, shape, size, and texture. After extraction by the alexnet, inception_v3, resnet34, and vgg13 models and selected by dimensionality reduction using principal components analysis (PCA), the deep-learning features were fused. Next, the gradient-weighted class-activation map (Grad-CAM) was used to visualize the-deep-learning process. Finally, the fusion features set, which was fused from the traditional omics features and the deep-fusion features, was used to establish the final classification models. The performance of the final models was evaluated by accuracy, confusion matrix, and the receiver operating characteristic (ROC) curve. Compared with other classification models, the performance of the support vector machine (SVM) model was best, with an accuracy of 93.8%. The area under curves AUC of micro- and macro-averages were 99%, and the AUC of the AMD, DME, RVO, and CSC groups were 100, 99, 98, and 100%, respectively. The artificial intelligence model in this study could be used to classify DME, AME, RVO, and CSC accurately from SD-OCT images.

Deep Multi-Input Multi-Stream Ordinal Model for age estimation: Based on spatial attention learning

Face aging process is non-stationary since human matures in different ways. This property makes age estimation is an attractive and challenging research topic in the computer vision community. Most of previous work conventionally estimate age from the center area of the aligned face image. However, these methods ignore spatial context information and cannot pay attention to particular domain features due to the uncertainty in deep learning. In this work, we propose a novel Deep Multi-Input Multi-Stream Ordinal (D2MO) Model for facial age estimation, which learns deep fusion feature through a specific spatial attention mechanism. Our approach is motivated by the observations that there are some universal changes, like hair color turning to white and wrinkles increasing, for individuals during aging process. In order to focus these spatial features, our D2MO uses four scales of receptive fields for global and contextual feature learning, and meanwhile, four cropped face patches are utilized for local and detailed feature extraction. Benefiting from a multi-stream CNN architecture, differentiated feature maps are learned separately through each branch and then aggregated together by concatenate layer. We also introduce a novel representation for age label using a multi-hot vector and the final predicted age can be calculated by summing the vector. This representation cast age estimation task to solve a series of binary classification subproblems which is easier to learn and more consistent with human cognition rather than to regress a single age value directly. Finally, we employ a joint training loss to supervise our model to learn the ordinal ranking, label distribution and regression information simultaneously. Extensive experiments show that our D2MO model significantly outperforms other state-of-the-art age estimation methods on MORPH II, FG-NET and UAGD datasets.

The intelligent detection method for flip chips using CBN-S-Net algorithm with SAM images

Flip chip has become one of the mainstream technologies in microelectronic packaging. Solder bumps play an important role in the interconnection of flip chips packages. The scanning acoustic microscopy (SAM) technology and a new network model were investigated for intelligent detection of flip chips. A new network model called CBN-S-Net was proposed based on a deep convolution network CBN and an optimized Siamese network. The CBN convolution network was used to extract the deep fusion features of solder bumps, and new triplet sample pairs were designed to measure the similarity between solder bumps. With the strategy of triplet sample pairs, the SAM images of the flip chip were used to verify the effectiveness of the designed network model. The results showed that the improved network has a high detection accuracy of 98.73%, and the proposed method is effective for the intelligent detection of solder bumps in high-density electronic packages.

A method of feature fusion and dimension reduction for knee joint pathology screening and separability evaluation criteria

Background and objectiveKnee-joint vibroarthrographic (VAG) signal is an effective method for performing a non-invasive knee osteoarthritis (KOA) diagnosis, VAG signal analysis plays a crucial role in achieving the early pathological screening of the knee joint. In order to improve the accuracy of knee pathology screening and to investigate the method suitable for embedded in wearable diagnostic device for knee joint, this paper proposes a knee pathology screening method. Aiming to fill the gap of lacking suitable and unified evaluation indexes for single feature and fusion feature, this paper proposes feature separability evaluation criteria. MethodsIn this paper, we propose a knee joint pathology screening method based on feature fusion and dimension reduction combined with random forest classifier, as well as, the evaluation criteria of feature separability. As for pathological screening method, this paper proposes the idea of multi-dimensional feature fusion, using principal component analysis (PCA) to reduce the redundant part of fusion feature (F-F) to obtain deep fusion feature (D-F-F) with more separability. Meanwhile, this paper proposes the maximal information coefficient (MIC) and correlation matrix collinearity (CMC) feature evaluation criteria, these not only can be used as new feature quantitative metrics, but also illustrate that the divisibility of the deep fusion feature is more potent than that before feature dimension reduction. ResultsThe experimental results show that the method in this paper has good performance in pathology classification on random forest classifier with 96% accuracy, especially the accuracy of SVM and K-NN are also improved after feature dimension reduction. ConclusionThe results indicate that this classification research has high screening efficiency for KOA diagnosis and could provide a feasible method for computer-assisted non-invasive diagnosis of KOA. And we provide a novel way for separability evaluation of VAG signal features.

A typhoon trajectory prediction model based on multimodal and multitask learning

Artificial intelligence technology has been widely used in various fields in recent years. In the case of typhoons, trajectory prediction technology can reduce the loss of human life and property caused by typhoon movements. From the perspective of deep learning, multimodal learning and multitask learning are applied to trajectory prediction. And a trajectory prediction model based on deep multimodal fusion and multitask generation (Trj-DMFMG) is proposed. The model mainly includes two modules: a deep multimodal fusion module and a multitask generation module. The deep multimodal fusion module is composed of several multimodal fusion modules. First, the multimodal trajectory sequence is divided into multiple multimodal subtrajectories by using a sliding window. Then, the multimodal fusion module trains different modal data to perform feature fusion through a long short-term memory network (LSTM) and a 3D convolution neural network (3D CNN). Finally, the features generated by multiple multimode fusion modules are deeply fused. The multitask generation module first trains the deep fusion features generated by the deep multimodal fusion module through the LSTM, then it realizes longitude and latitude prediction at the same time. In this paper, real typhoon data in the Northwest Pacific Ocean are used for simulation experiments. Through a comprehensive comparison of the prediction results in longitude and latitude, it is found that Trj-DMFMG has the best prediction effect and is more accurate and stable in long-term prediction.

Hyperspectral Remote Sensing Images Deep Feature Extraction Based on Mixed Feature and Convolutional Neural Networks

To achieve effective deep fusion features for improving the classification accuracy of hyperspectral remote sensing images (HRSIs), a pixel frequency spectrum feature is presented and introduced to convolutional neural networks (CNNs). Firstly, the fast Fourier transform is performed on each spectral pixel to obtain the amplitude spectrum, i.e., the pixel frequency spectrum feature. Then, the obtained pixel frequency spectrum is combined with the spectral pixel to form a mixed feature, i.e., spectral and frequency spectrum mixed feature (SFMF). Several multi-branch CNNs fed with pixel frequency spectrum, SFMF, spectral pixel, and spatial features are designed for extracting deep fusion features. A pre-learning strategy, i.e., basic single branch CNNs are used to pre-learn the weights of a multi-branch CNN, is also presented for improving the network convergence speed and avoiding the network from getting into a locally optimal solution to a certain extent. And after reducing the dimensionality of SFMF by principal component analysis (PCA), a 3-dimensionality (3-D) CNN is also designed to further extract the joint spatial-SFMF feature. The experimental results of three real HRSIs show that adding the presented frequency spectrum feature into CNNs can achieve better recognition results, which in turn proves that the presented multi-branch CNNs can obtain the deep fusion features with more discriminant information.

Computer-Aided Intracranial EEG Signal Identification Method Based on a Multi-Branch Deep Learning Fusion Model and Clinical Validation.

Surgical intervention or the control of drug-refractory epilepsy requires accurate analysis of invasive inspection intracranial EEG (iEEG) data. A multi-branch deep learning fusion model is proposed to identify epileptogenic signals from the epileptogenic area of the brain. The classical approach extracts multi-domain signal wave features to construct a time-series feature sequence and then abstracts it through the bi-directional long short-term memory attention machine (Bi-LSTM-AM) classifier. The deep learning approach uses raw time-series signals to build a one-dimensional convolutional neural network (1D-CNN) to achieve end-to-end deep feature extraction and signal detection. These two branches are integrated to obtain deep fusion features and results. Resampling is employed to split the imbalanced epileptogenic and non-epileptogenic samples into balanced subsets for clinical validation. The model is validated over two publicly available benchmark iEEG databases to verify its effectiveness on a private, large-scale, clinical stereo EEG database. The model achieves high sensitivity (97.78%), accuracy (97.60%), and specificity (97.42%) on the Bern–Barcelona database, surpassing the performance of existing state-of-the-art techniques. It is then demonstrated on a clinical dataset with an average intra-subject accuracy of 92.53% and cross-subject accuracy of 88.03%. The results suggest that the proposed method is a valuable and extremely robust approach to help researchers and clinicians develop an automated method to identify the source of iEEG signals.

Remote Sensing Image Recognition Based on Multi-attention Residual Fusion Networks

Since each sample in a hyperspectral remote sensing image is made up of high-dimensional features and contains a wealth of remote sensing features, feature selection and mining become more difficult. To address this issue, a multi-attention residual integrated network (MARB-Net) algorithm is proposed, which reduces redundant features while increasing feature fusion and, as a result, improves hyperspectral image recognition. First, assign multiple weights to each feature using multiple attention mechanism models; then, deep mine and integrate the features using the residual network; and finally, perform contextual semantic integration on the deep fusion features using the Bi-LSTM network. The recognition task should be completed by the Softmax classifier. The experimental results on three multi-class public data sets show that the MARB-Net algorithm proposed in this paper is effective.

Non Rigid 3D Shape Partial Matching Based on Deep Feature Fusion

<p indent=0mm>To meet the requirements of massive and heterogeneous 3D shape partial matching and intelligent retrieval technology, a 3D shape local matching method based on the deep fusion feature of F-PointCNN is proposed. First, the feature bag (BoF) learning model is used to propose an geometric image representation, which can not only effectively distinguish heterogeneous non-rigid 3D models of the same kind, but also reveal the structural similarity of large-scale incomplete 3D models. Next, a cascaded convolutional neural network slearning framework (F-PointCNN) is constructed, where BoF-CNN learns the deep global feature from BoF geometric images and establishes the point feature representation that integrates the local feature and the global feature; Point-CNN refines the point feature and generates deep feature representation which effectively improves the discriminative ability and robustness. Finally, the local shape matching of non rigid 3D model is realized by cross matrix measurement. The open non-rigid 3D shape databases are used to carry out a series of experiments, the results show that the features extracted by proposed method have stronger discriminative ability in large-scale transformation shape classification and higher precision in partial shape matching.

Deep Fusion Feature Extraction for Caries Detection on Dental Panoramic Radiographs

Caries is the most well-known disease and relates to the oral health of billions of people around the world. Despite the importance and necessity of a well-designed detection method, studies in caries detection are still limited and show a restriction in performance. In this paper, we proposed a computer-aided diagnosis (CAD) method to detect caries among normal patients using dental radiographs. The proposed method mainly consists of two processes: feature extraction and classification. In the feature extraction phase, the chosen 2D tooth image was employed to extract deep activated features using a deep pre-trained model and geometric features using mathematic formulas. Both feature sets were then combined, called fusion feature, to complement each other defects. Then, the optimal fusion feature set was fed into well-known classification models such as support vector machine (SVM), k-nearest neighbor (KNN), decision tree (DT), Naïve Bayes (NB), and random forest (RF) to determine the best classification model that fit the fusion features set and perform the most preeminent result. The results show 91.70%, 90.43%, and 92.67% for accuracy, sensitivity, and specificity, respectively. The proposed method has outperformed the previous state-of-the-art and shows promising results when none of the measured factors is less than 90%; therefore, the method is promising for dentists and capable of wide-scale implementation caries detection in hospitals.

TARDB-Net: triple-attention guided residual dense and BiLSTM networks for hyperspectral image classification

Each sample in the hyperspectral remote sensing image has high-dimensional features and contains rich spatial and spectral information, which greatly increases the difficulty of feature selection and mining. In view of these difficulties, we propose a novel Triple-attention Guided Residual Dense and BiLSTM networks(TARDB-Net) to reduce redundant features while increasing feature fusion capabilities, which ultimately improves the ability to classify hyperspectral images. First, a novel Triple-attention mechanism is proposed to assign different weights to each feature. Then, the residual network is used to perform the residual operation on the features, and the initial features of the multiple residual blocks and the generated deep residual features are intensively fused, retaining a host number of prior features. And use the bidirectional long short-term memory network to integrate the contextual semantics of deep fusion features. Finally, the classification task is completed by Softmax classifier. Experiments on three hyperspectral datasets—Indian Pines, University of Pavia, and Salinas—show that under 10% of the training samples, the overall accuracy of our method is 87%, 96% and 96%, which is superior to several well-known methods.

Research on Modeling of Vocal State Duration Based on Spectrogram Analysis

In the early stage of vocal music education, students generally do not understand the structure of the human body, and have doubts about how to pronounce their voices scientifically. However, with the continuous development of computers, computer technology has become more and more developed, and computer processing speed has been greatly increased, which provides favorable conditions for the development of the application of vocal spectrum analysis technology in vocal music teaching. In this paper, we first study the GMM-SVM and DBN, and combine them to extract the deep Gaussian super vector DGS, and further construct the feature DGCS on the basis of DGS; then we study the convolutional neural network (CNN), which has achieved great success in the image recognition task in recent years, and design a CNN model to extract the deep fusion features of vocal music. The experimental simulations show that the CNN fusion-based speaker recognition system achieves very good results in terms of recognition rate.

A Deep Fusion Gaussian Mixture Model for Multiview Land Data Clustering

With the rapid industrialization and urbanization, pattern mining of soil contamination of heavy metals is attracting increasing attention to control soil contamination. However, the correlation over various heavy metals and the high-dimension representation of heavy metal data pose vast challenges on the accurate mining of patterns over heavy metals of soil contamination. To solve those challenges, a multiview Gaussian mixture model is proposed in this paper, to naturally capture complicated relationships over multiviews on the basis of deep fusion features of data. Specifically, a deep fusion feature architecture containing modality-specific and modality-common stacked autoencoders is designed to distill fusion representations from the information of all views. Then, the Gaussian mixture model is extended on the fusion representations to naturally recognize the accurate patterns of the intra- and inter-views. Finally, extensive experiments are conducted on the representative datasets to evaluate the performance of the multiview Gaussian mixture model. Results show the outperformance of the proposed methods.

Deep Fusion Feature Representation Learning With Hard Mining Center-Triplet Loss for Person Re-Identification

Person re-identification (Re-ID) is a challenging task in the field of computer vision and focuses on matching people across images from different cameras. The extraction of robust feature representations from pedestrian images through CNNs with a single deterministic pooling operation is problematic as the features in real pedestrian images are complex and diverse. To address this problem, we propose a novel center-triplet (CT) model that combines the learning of robust feature representation and the optimization of metric loss function. Firstly, we design a fusion feature learning network (FFLN) with a novel fusion strategy consisting of max pooling and average pooling. Instead of adopting a single deterministic pooling operation, the FFLN combines two pooling operations that can learn high response values, bright features, and low response values, discriminative features simultaneously. Our model obtains more discriminative fusion features by adaptively learning the weights of the features learned by the corresponding pooling operations. In addition, we design a hard mining center-triplet loss (HCTL), a novel improved triplet loss, which effectively optimizes the intra/inter-class distance and reduces the cost of computing and mining hard training samples simultaneously, thereby enhancing the learning of robust feature representation. Finally, we proved our method can learn robust and discriminative feature representations for complex pedestrian images in real scenes. The experimental results also illustrate that our method achieves an 81.8% mAP and a 93.8% rank-1 accuracy on Market1501, a 68.2% mAP and an 83.3% rank-1 accuracy on DukeMTMC-ReID, and a 43.6% mAP and a 74.3% rank-1 accuracy on MSMT17, outperforming most state-of-the-art methods and achieving better performance for person re-identification.