Classifier Fusion Research Articles

Binaural Acoustic Scene Classification Using Wavelet Scattering, Parallel Ensemble Classifiers and Nonlinear Fusion.

The analysis of ambient sounds can be very useful when developing sound base intelligent systems. Acoustic scene classification (ASC) is defined as identifying the area of a recorded sound or clip among some predefined scenes. ASC has huge potential to be used in urban sound event classification systems. This research presents a hybrid method that includes a novel mathematical fusion step which aims to tackle the challenges of ASC accuracy and adaptability of current state-of-the-art models. The proposed method uses a stereo signal, two ensemble classifiers (random subspace), and a novel mathematical fusion step. In the proposed method, a stable, invariant signal representation of the stereo signal is built using Wavelet Scattering Transform (WST). For each mono, i.e., left and right, channel, a different random subspace classifier is trained using WST. A novel mathematical formula for fusion step was developed, its parameters being found using a Genetic algorithm. The results on the DCASE 2017 dataset showed that the proposed method has higher classification accuracy (about 95%), pushing the boundaries of existing methods.

An FPGA-Based Upper-Limb Rehabilitation Device for Gesture Recognition and Motion Evaluation Using Multi-Task Recurrent Neural Networks

Upper-Extremity motor impairment affects millions of Americans due to cerebrovascular incidents, spinal cord injuries, or brain trauma. Current therapy practices used to assist these individuals in regaining motor functionality often require extensive time at rehabilitation facilities with potentially prohibitive travel or financial costs. This work presents a mobile low-cost field programmable gate array (FPGA)-smart rehabilitation system that can be used in home environments. The prototype is a rehabilitation table instrumented with a capacitive sensor array (CSA) to track upper-extremity motions of the user through proximity or touch. In addition, inertial measurement units (IMUs) are placed on the affected upper limb and combined with the CSA data with our sensor fusion signal processing architecture. Motions are classified and evaluated using multi-task convolutional recurrent neural networks with three additional motion quality output classes to personalize recognition based on the particular motor skills of each patient. The prototype achieves above 99% accuracy with 32-bit fixed-point format implementation for recognizing dynamic motions and identifying unnatural characteristics (i.e., tremor or limited flexion and extension) in upper limb motions based on sensor values. The convolutional recurrent neural network (C-RNN) fusion classification network is implemented on a 200 MHz Zynq ZCU104 FPGA using an HLS-based design optimized with pipelining and parallelism techniques and achieves 5.4x speedup compared to ARM® Cortex-A53 implementation running at an operating frequency of 1.3 GHz. The prototype is also demonstrated to perform the machine learning classification in real-time.

Fusion Classification of HSI and MSI Using a Spatial-Spectral Vision Transformer for Wetland Biodiversity Estimation

The rapid development of remote sensing technology provides wealthy data for earth observation. Land-cover mapping indirectly achieves biodiversity estimation at a coarse scale. Therefore, accurate land-cover mapping is the precondition of biodiversity estimation. However, the environment of the wetlands is complex, and the vegetation is mixed and patchy, so the land-cover recognition based on remote sensing is full of challenges. This paper constructs a systematic framework for multisource remote sensing image processing. Firstly, the hyperspectral image (HSI) and multispectral image (MSI) are fused by the CNN-based method to obtain the fused image with high spatial-spectral resolution. Secondly, considering the sequentiality of spatial distribution and spectral response, the spatial-spectral vision transformer (SSViT) is designed to extract sequential relationships from the fused images. After that, an external attention module is utilized for feature integration, and then the pixel-wise prediction is achieved for land-cover mapping. Finally, land-cover mapping and benthos data at the sites are analyzed consistently to reveal the distribution rule of benthos. Experiments on ZiYuan1-02D data of the Yellow River estuary wetland are conducted to demonstrate the effectiveness of the proposed framework compared with several related methods.

Directionally separable dilated CNN with hierarchical attention feature fusion for hyperspectral image classification

ABSTRACT In recent years, the convolutional neural network (CNN) plays a vital role in hyperspectral image classification and performs more competitively than many other methods. However, in order to pursue better performance, most of existing CNN-based methods just simply stack rather deep convolutional layers. Although they improve the classification accuracy to a certain extent, they result in plenty of network parameters. In this paper, a light-weighted directionally separable dilated CNN with hierarchical attention feature fusion (DSD-HAFF) is proposed to solve these problems. First, two global dense dilated CNN branches that focus on two spatial directions separately are constructed to extract and reuse spatial information as much as possible. Second, a hierarchical attention feature fusion branch that consists of several coordinate attention blocks (CABs) is constructed. Hierarchical features from two directionally separable dilated CNN branches are adopted as inputs of CABs. In this way, the structure can not only fully incorporate hierarchical features, but also significantly reduce the network parameters. Meanwhile, the hierarchical attention feature fusion branch incorporates features from high-level to low-level in the kernel-number pyramid strategy. Experimental results on three popular benchmark datasets demonstrate that the DSD-HAFF achieves better performance and has a much smaller number of network parameters than the other state-of-the-art methods.

Binary imbalanced big data classification based on fuzzy data reduction and classifier fusion

Binary imbalanced big data classification based on fuzzy data reduction and classifier fusion

Hierarchy-aware contrastive learning with late fusion for skin lesion classification

Background and ObjectiveThe incidence rate of skin cancers is increasing worldwide annually. Using machine learning and deep learning for skin lesion classification is one of the essential research topics. In this study, we formulate a major-type misclassification problem that previous studies did not consider in the multi-class skin lesion classification. Moreover, addressing the major-type misclassification problem is significant for real-world computer-aided diagnosis. MethodsThis study presents a novel method, namely Hierarchy-Aware Contrastive Learning with Late Fusion (HAC-LF), to improve the overall performance of multi-class skin classification. In HAC-LF, we design a new loss function, Hierarchy-Aware Contrastive Loss (HAC Loss), to reduce the impact of the major-type misclassification problem. The late fusion method is applied to balance the major-type and multi-class classification performance. ResultsWe conduct a series of experiments with the ISIC 2019 Challenges dataset, which consists of three skin lesion datasets, to verify the performance of our methods. The results show that our proposed method surpasses the representative deep learning methods for skin lesion classification in all evaluation metrics used in this study. HAC-LF achieves 0.871, 0.842, 0.889 for accuracy, sensitivity, and specificity in the major-type classification, respectively. With the imbalanced class distribution, HAC-LF outperforms the baseline model regarding the sensitivity of minority classes. ConclusionsThis research formulates a major-type misclassification problem. We propose HAC-LF to deal with it and boost the multi-class skin lesion classification performance. According to the results, the advantage of HAC-LF is that the proposed HAC Loss can beneficially reduce the impact of the major-type misclassification by decreasing the major-type error rate. Besides the medical field HAC-LF is promising to be applied to other domains possessing the data with the hierarchical structure.

Deep manifold feature fusion for classification of breast histopathology images

Histopathology image analysis is the gold standard for breast cancer diagnosis, yet the classification of breast histopathology images is challenging. Convolutional neural networks usually classify images based on deep abstract features only, ignoring the influence of structure information contained in low-level features, which limits the classification ability of deep models. To address the above problem, a breast histopathology image classification method based on the deep manifold fusion of multilayer features is proposed, LPMF2Net. By exploring the complementarity between features at different levels, the multilayer features were cross-cascaded and fused to enhance the cell structure characterization ability. Local preserving projection is applied for the fused features to reduce the interference of redundant information and optimize fusion performance. Moreover, the projection matrix was adaptively adjusted according to the local preserving regularization term to further optimize the model. The proposed LPMF2Net model was tested on the public dataset BreaKHis, and the experimental results (40×: 94.91%, 100×: 96.12%, 200×: 95.51%, 400×: 95.42%) proved its effectiveness.

Dilated Multi-scale Fusion for Point Cloud Classification and Segmentation

Dilated Multi-scale Fusion for Point Cloud Classification and Segmentation

Multi-Modal Feature Selection with Feature Correlation and Feature Structure Fusion for MCI and AD Classification.

Feature selection for multiple types of data has been widely applied in mild cognitive impairment (MCI) and Alzheimer’s disease (AD) classification research. Combining multi-modal data for classification can better realize the complementarity of valuable information. In order to improve the classification performance of feature selection on multi-modal data, we propose a multi-modal feature selection algorithm using feature correlation and feature structure fusion (FC2FS). First, we construct feature correlation regularization by fusing a similarity matrix between multi-modal feature nodes. Then, based on manifold learning, we employ feature matrix fusion to construct feature structure regularization, and learn the local geometric structure of the feature nodes. Finally, the two regularizations are embedded in a multi-task learning model that introduces low-rank constraint, the multi-modal features are selected, and the final features are linearly fused and input into a support vector machine (SVM) for classification. Different controlled experiments were set to verify the validity of the proposed method, which was applied to MCI and AD classification. The accuracy of normal controls versus Alzheimer’s disease, normal controls versus late mild cognitive impairment, normal controls versus early mild cognitive impairment, and early mild cognitive impairment versus late mild cognitive impairment achieve 91.85 ± 1.42%, 85.33 ± 2.22%, 78.29 ± 2.20%, and 77.67 ± 1.65%, respectively. This method makes up for the shortcomings of the traditional multi-modal feature selection based on subjects and fully considers the relationship between feature nodes and the local geometric structure of feature space. Our study not only enhances the interpretation of feature selection but also improves the classification performance, which has certain reference values for the identification of MCI and AD.

A rapid, low-cost deep learning system to classify strawberry disease based on cloud service

Accurate and timely classification of diseases during strawberry planting can help growers deal with them in timely manner, thereby reducing losses. However, the classification of strawberry diseases in real planting environments is facing severe challenges, including complex planting environments, multiple disease categories with small differences, and so on. Although recent mobile vision technology based deep learning has achieved some success in overcoming the above problems, a key problem is how to construct a non-destructive, fast and convenient method to improve the efficiency of strawberry disease identification for the multi-region, multi-space and multi-time classification requirements. We develop and evaluate a rapid, low-cost system for classifying diseases in strawberry cultivation. This involves designing an easy-to-use cloud-based strawberry disease identification system, combined with our novel self-supervised multi-network fusion classification model, which consists of a Location network, a Feedback network and a Classification network to identify the categories of common strawberry diseases. With the help of a novel self-supervision mechanism, the model can effectively identify diseased regions of strawberry disease images without the need for annotations such as bounding boxes. Using accuracy, precision, recall and F1 to evaluate the classification effect, the results of the test set are 92.48, 90.68, 86.32 and 88.45%, respectively. Compared with popular Convolutional Neural Networks (CNN) and five other methods, our network achieves better disease classification effect. Currently, the client (mini program) has been released on the WeChat platform. The mini program has perfect classification effect in the actual test, which verifies the feasibility and effectiveness of the system, and can provide a reference for the intelligent research and application of strawberry disease identification.

Classifier Fusion for Detection of COVID-19 from CT Scans.

The coronavirus disease (COVID-19) is an infectious disease caused by the SARS-CoV-2 virus. COVID-19 is found to be the most infectious disease in last few decades. This disease has infected millions of people worldwide. The inadequate availability and the limited sensitivity of the testing kits have motivated the clinicians and the scientist to use Computer Tomography (CT) scans to screen COVID-19. Recent advances in technology and the availability of deep learning approaches have proved to be very promising in detecting COVID-19 with increased accuracy. However, deep learning approaches require a huge labeled training dataset, and the current availability of benchmark COVID-19 data is still small. For the limited training data scenario, the CNN usually overfits after several iterations. Hence, in this work, we have investigated different pre-trained network architectures with transfer learning for COVID-19 detection that can work even on a small medical imaging dataset. Various variants of the pre-trained ResNet model, namely ResNet18, ResNet50, and ResNet101, are investigated in the current paper for the detection of COVID-19. The experimental results reveal that transfer learned ResNet50 model outperformed other models by achieving a recall of 98.80% and an F1-score of 98.41%. To further improvise the results, the activations from different layers of best performing model are also explored for the detection using the support vector machine, logistic regression and K-nearest neighbor classifiers. Moreover, a classifier fusion strategy is also proposed that fuses the predictions from the different classifiers via majority voting. Experimental results reveal that via using learned image features and classification fusion strategy, the recall, and F1-score have improvised to 99.20% and 99.40%.

Tropical Cyclones Intensity Estimation by Feature Fusion and Random Forest Classifier Using Satellite Images

Tropical Cyclones Intensity Estimation by Feature Fusion and Random Forest Classifier Using Satellite Images

Squeeze-and-Excitation Laplacian Pyramid Network With Dual-Polarization Feature Fusion for Ship Classification in SAR Images

This letter proposes a squeeze-and-excitation Laplacian pyramid network with dual-polarization feature fusion (SE-LPN-DPFF) for ship classification in synthetic aperture radar (SAR) images. SE-LPN-DPFF offers three contributions: 1) dual-polarization (VV and VH) feature fusion (DPFF); 2) channel modeling by the squeeze-and-excitation (SE) to balance each polarization feature’s contribution; and 3) Laplacian pyramid network (LPN) to achieve multiresolution analysis (MRA). Extensive ablation studies can confirm the effectiveness of each contribution. Results on the three- and six-category OpenSARShip datasets reveal the state-of-the-art SAR ship classification performance.

HOG-ShipCLSNet: A Novel Deep Learning Network With HOG Feature Fusion for SAR Ship Classification

Ship classification in synthetic aperture radar (SAR) images is a fundamental and significant step in ocean surveillance. Recently, with the rise of deep learning (DL), modern abstract features from convolutional neural networks (CNNs) have hugely improved SAR ship classification accuracy. However, most existing CNN-based SAR ship classifiers overly rely on abstract features, but uncritically abandon traditional mature hand-crafted features, which may incur some challenges for further improving accuracy. Hence, this article proposes a novel DL network with histogram of oriented gradient (HOG) feature fusion (HOG-ShipCLSNet) for preferable SAR ship classification. In HOG-ShipCLSNet, four mechanisms are proposed to ensure superior classification accuracy, that is, 1) a multiscale classification mechanism (MS-CLS-Mechanism); 2) a global self-attention mechanism (GS-ATT-Mechanism); 3) a fully connected balance mechanism (FC-BAL-Mechanism); and 4) an HOG feature fusion mechanism (HOG-FF-Mechanism). We perform sufficient ablation studies to confirm the effectiveness of these four mechanisms. Finally, our experimental results on two open SAR ship datasets (OpenSARShip and FUSAR-Ship) jointly reveal that HOG-ShipCLSNet dramatically outperforms both modern CNN-based methods and traditional hand-crafted feature methods.

MSLAENet: Multiscale Learning and Attention Enhancement Network for Fusion Classification of Hyperspectral and LiDAR Data

The effective use of multimodal data to obtain accurate land cover information has become an interesting and challenging research topic in the field of remote sensing. In this paper, we propose a new method, multi-scale learning and attention enhancement network (MSLAENet), to implement Hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion classification in an end-to-end manner. Specifically, our model consists of three main modules. First, we design the composite attention (CA) module, which adopts self-attention to enhance the feature representations of HSI and LiDAR data, respectively, and cross-attention to achieve cross-modal information enhancement. Second, the proposed multi-scale learning (MSL) module combines self-calibrated convolutions and hierarchical residual structure to extract different scales of information to further improve the representation capability of the model. Finally, the attention-based feature fusion (FF) module fully considers the complementary information properties between different modalities and adaptively fuses heterogeneous features from different modalities. To test the performance of MSLAENet, we conduct experiments on three multimodal remote sensing datasets and compare them with the state-of-the-art fusion model, which demonstrates the effectiveness and superiority of the model.

SAR Target Classification Using the Multikernel-Size Feature Fusion-Based Convolutional Neural Network

It is well-known that the convolutional neural network (CNN) is an effective method for synthetic aperture radar (SAR) target classification. In the convolutional layer of CNN, convolutional kernels of different sizes can extract different feature information of the target. The small-size kernel can extract the local texture feature information, and the large-size kernel can extract the global contour feature information. Traditional CNN methods usually use fixed-size kernels for convolution, and they generally lose part of the target’s feature information, resulting in the inaccurate classification of the SAR targets. This article proposes a novel CNN model based on multikernel-size feature fusion (MKSFF-CNN) for SAR target classification. MKSFF-CNN designs a convolutional methodology with a multichannel parallel topology, it uses convolutional kernels of different sizes to extract the multikernel-size deep features of the SAR target, and then, these features are fused in an optimal way to acquire the lowest loss. Moreover, MKSFF-CNN concatenates the fused features extracted by the convolutional layers of different dimensions to achieve the finest classification. MKSFF-CNN greatly elevates the feature representation completeness of the SAR targets so that more useful feature information can be exploited for SAR target classification. Undoubtedly, MKSFF-CNN can achieve a better classification performance compared with traditional CNN models with a fixed kernel size. The superiority of MKSFF-CNN is validated on the moving and stationary target acquisition and recognition (MSTAR) dataset with the detailed objective and subjective evaluation.

A Fine PolSAR Terrain Classification Algorithm Using the Texture Feature Fusion-Based Improved Convolutional Autoencoder

In order to more efficiently mine the features of polarimetric synthetic aperture radar (PolSAR) and establish a more appropriate classification model, this article proposes an improved convolutional autoencoder (ICAE) based on texture feature fusion (TFF-ICAE) for PolSAR terrain classification. First, TFF-ICAE specifically designs a multi-indicator squeeze-and-excitation (MI-SE) block and incorporates it into the CAE network. MI-SE can enhance the essential feature information while suppressing the interference information as much as possible, and it can effectively increase the between-class distance while reducing the within-class distance. Then, TFF-ICAE uses gray level co-occurrence matrix (GLCM) to capture the texture features, and it optimally fuses these texture features and the deep features extracted by ICAE to complete the multilevel feature fusion, elevating the feature representation completeness of the terrain. That is, TFF-ICAE effectively enhances the feature separation capability of different categories while greatly elevating the feature representation completeness. Experiments on the datasets of San Francisco, Oberpfaffenhofen, and Flevoland show that the proposed TFF-ICAE, respectively, achieves overall accuracies of 93.44%, 97.61%, and 97.78%, which are at least 0.92%, 1.52%, and 0.97% higher than other algorithms. Undoubtedly, the superiority of TFF-ICAE is verified on these datasets.

Novel Cross-Resolution Feature-Level Fusion for Joint Classification of Multispectral and Panchromatic Remote Sensing Images

With the increasing availability and resolution of satellite sensor data, multispectral (MS) and panchromatic (PAN) images are the most popular data that are used in remote sensing among applications. This article proposes a novel cross-resolution hidden layer feature fusion (CRHFF) approach for joint classification of multiresolution MS and PAN images. In particular, shallow spectral and spatial features at a global scale are first extracted from an MS image. Then, deep cross-resolution hidden layer features extracted from MS and PAN are fused from patches at a local scale according to an autoencoder (AE)-like deep network. Finally, the selected multiresolution hidden layer features are classified in a supervised manner. By taking advantage of integrated shallow-to-deep and global-to-local features from the high-resolution MS and PAN images, the cross-resolution latent information can be extracted and fused in order to better model imaged objects from the multimodal representation and finally increase the classification accuracy. Experimental results obtained on three real multiresolution datasets covering complex urban scenarios confirm the effectiveness of the proposed approach in terms of higher accuracy and robustness with respect to literature methods.

Spectral and Spatial Feature Fusion for Hyperspectral Image Classification

Compared with traditional images, hyperspectral images (HSI) not only have spatial information, but also have rich spectral information. However, the mainstream hyperspectral image classification (HIC) methods are all based on Convolutional Neural Network (CNN), which has great advantages in extracting spatial features, but it has certain limitations in dealing with spectral continuous sequence information. Therefore Transformer which is good at processing sequences, has also been gradually applied to HIC. Besides, Since HSI are typical three-dimensional structures, we believe that the correlation of the three dimensions is also an important information. So in order to fully extract the spectral spatial information, as well as the correlation of the three dimensions. we propose a spectral and spatial feature fusion module ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., TransCNN) for HIC. TransCNN consists of CNNs and a Transformer. The former is in charge of mining the spatial and spectral information from different dimensions, while the latter not only undertakes the most critical fusion but also captures the deeper relationship characteristics. We transpose the data to extract features and their correlation through three CNNs branches. we believe that these feature maps still have deep spectral information. Therefore, we have embedded them into one-dimensional vectors and use Transformer’s Encoder to extract features. However, some information will be lost when embedding into one-dimensional vectors. Therefore we use Decoder, which has been ignored in the field of vision, to fuse the features before passing Encoder and the features after extracted by Encoders. Two kinds of features are fused by Decoder, and the obtained information is finally input into the classifier for classification. Experimental results on real HSIs show that the proposed architecture can achieve competitive performance compared with the state-of-the-art methods.

Multiple Models Fusion for Multi-label Classification in Speech Emotion Recognition Systems

Emotions are the base to comprehend speech and respond to each other. However, interpreting emotions is not a simple process even for people, not to mention machines. Based on a ground truth investigation that we have done on the RML (Ryerson Multimedia Lab) dataset, we have found that the same emotional speech might be interpreted differently by a number of individuals despite it has only one label. In this study we provide a fresh perspective to the speech emotion recognition field where a system may identify the presence of one or more potential emotions per utterance. Giving the computer the potential of identifying numerous categories at the same time, will allow it to assess different scenarios when making a decision. In this paper, we introduce a robust Speech Emotion Recognition (SER) system for multi-label classification. The model was tested on the RML and it achieved 83.88% which outperforms state-of-the-art methods.