Heterogeneous Feature Fusion Research Articles

Features fusion of multichannel wrist pulse signal based on KL-MGDCCA and decision level combination

To sense the pulse in the representative positions of wrist is the basis of traditional Chinese pulse diagnosis. The pulse diagnosis has been obtaining more and more attentions for its non-invasive character and its convenience in analysis of health status. For objective analysis, various types of pulse features have been extracted from the pulse signal with the development of computerized pulse analysis. The effective utilization of the features of multichannel is the increasing and urgent need for the pulse analysis. A novel features fusion frame is proposed to reduce the redundant information for the homogeneous features, and eliminate the interference of heterogeneous features. For the same type of features extracted from the different channels, the proposed method uses Karhunen–Loeve multiple generalized discriminative canonical correlation analysis (KL-MGDCCA) to fuse them into one feature vector. A support vector machine (SVM) classifier is trained for each type of fused features. Then, the frame adopts decision level fusion approach to combine these classifiers for pulse signal classification to solve the problem of heterogeneous features fusion. Extensive experiments show that the proposed fusion frame can achieve the best performance on most indicators for multichannel pulse signal analysis. For the classification of Diabetes/Health, Nephropathy/Health and Diabetes/Nephropathy, the proposed method achieves the best F-score with the value of 75.25%, 79.02% and 56.54%, which outperforms state-of-the-art methods being compared.

Discrete Multi-Graph Clustering.

Spectral clustering plays a significant role in applications that rely on multi-view data due to its well-defined mathematical framework and excellent performance on arbitrarily-shaped clusters. Unfortunately, directly optimizing the spectral clustering inevitably results in an NP-hard problem due to the discrete constraints on the clustering labels. Hence, conventional approaches intuitively include a relax-and-discretize strategy to approximate the original solution. However, there are no principles in this strategy that prevent the possibility of information loss between each stage of the process. This uncertainty is aggravated when a procedure of heterogeneous features fusion has to be included in multi-view spectral clustering. In this paper, we avoid an NP-hard optimization problem and develop a general framework for multi-view discrete graph clustering by directly learning a consensus partition across multiple views, instead of using the relax-and-discretize strategy. An effective re-weighting optimization algorithm is exploited to solve the proposed challenging problem. Further, we provide a theoretical analysis of the model's convergence properties and computational complexity for the proposed algorithm. Extensive experiments on several benchmark datasets verify the effectiveness and superiority of the proposed algorithm on clustering and image segmentation tasks.

Advanced Heterogeneous Feature Fusion Machine Learning Models and Algorithms for Improving Indoor Localization.

In the era of the Internet of Things and Artificial Intelligence, the Wi-Fi fingerprinting-based indoor positioning system (IPS) has been recognized as the most promising IPS for various applications. Fingerprinting-based algorithms critically rely on a fingerprint database built from machine learning methods. However, currently methods are based on single-feature Received Signal Strength (RSS), which is extremely unstable in performance in terms of precision and robustness. The reason for this is that single feature machines cannot capture the complete channel characteristics and are susceptible to interference. The objective of this paper is to exploit the Time of Arrival (TOA) feature and propose a heterogeneous features fusion model to enhance the precision and robustness of indoor positioning. Several challenges are addressed: (1) machine learning models based on heterogeneous features, (2) the optimization of algorithms for high precision and robustness, and (3) computational complexity. This paper provides several heterogeneous features fusion-based localization models. Their effectiveness and efficiency are thoroughly compared with state-of-the-art methods.

Context Semantics for Small Target Detection in Large-Field Images with Two Cascaded Faster R-CNNs

Computer vision and image processing techniques have been widely applied to power transmission line inspection. However, the successful detection of small targets in large scenes is still challenging due to their low resolution and poor feature representation. Existing methods, such as multi-scale image pyramid, multi-scale feature pyramid and multiple heterogeneous feature fusion, etc. can extract more representative features of small objects, but they usually require high computation cost. In this paper, we propose an effective two cascaded Faster R-CNN strategy, which is based on multi-scale features and semantic information between the objects and the background, to address the small target detection in large scenes. Specially, we detect large object candidate proposals that may contain small objects at first and then map them to the original images to detect the small-sized targets on the high resolution regions. Experiments show that our strategy could lead to higher (83.0%) accuracy of small target detection and recognition than the one-stage Faster R-CNN (78.3%) on the dataset of aerial images.

A deep heterogeneous feature fusion approach for automatic land-use classification

In remote sensing, images have assorted appearances due to fused boundaries, making it difficult to generate discriminative feature representation for the purpose of conducting classification tasks. Therefore, an effective feature representation can boost the classification accuracy in the field of satellite imaging. In this paper, we propose a novel hybrid system for satellite image classification that combines the distinct information of deep features, and generate a discriminative representation by preserving the essential information of original feature space. We use pre-trained convolutional neural networks for extracting our features via transfer learning. For this purpose, we first propose a single strategy where fully connected layers are effectively used to represent different levels of image features. Secondly, a robust approach, called entropy controlled neighborhood component analysis, is then proposed to optimize fusion of multiple layers of different architectures in a unified hierarchical manner. To validate the effectiveness of the proposed approach, we perform experiments on three benchmark satellite datasets; UC MERCED, RS19 and AID. We statistically analyze our results with analysis of variance and post-hoc Bonferroni test, and compare our proposed methodology with state-of-the-art methods. Experimental results show that the proposed methodology can accurately classify satellite images with 99.7%, 99.1% and 92.2% accuracy with selected classifier and by utilizing less than 5% features.

Integration of heterogeneous features for remote sensing scene classification

Scene classification is one of the most important issues in remote sensing (RS) image processing. We find that features from different channels (shape, spectral, texture, etc.), levels (low-level and middle-level), or perspectives (local and global) could provide various properties for RS images, and then propose a heterogeneous feature framework to extract and integrate heterogeneous features with different types for RS scene classification. The proposed method is composed of three modules (1) heterogeneous features extraction, where three heterogeneous feature types, called DS-SURF-LLC, mean-Std-LLC, and MS-CLBP, are calculated, (2) heterogeneous features fusion, where the multiple kernel learning (MKL) is utilized to integrate the heterogeneous features, and (3) an MKL support vector machine classifier for RS scene classification. The proposed method is extensively evaluated on three challenging benchmark datasets (a 6-class dataset, a 12-class dataset, and a 21-class dataset), and the experimental results show that the proposed method leads to good classification performance. It produces good informative features to describe the RS image scenes. Moreover, the integration of heterogeneous features outperforms some state-of-the-art features on RS scene classification tasks.

Heterogeneous massive feature fusion on grassmannian manifold

Two issues remain unsolved on utilizing multimodal features for pattern recognition: the missing features and the curse of dimensionality. In this paper, we address the two issues by fusing the multimodal features on the Grassmann manifold. By defining grouping constrains on multimodal features, each multimodal feature vector is grouped into a set of subspaces, and is further represented as a point on the Grassmann manifold. To deal with missing features, L2-Hausdorff distance, a metric to compare multimodal feature vectors with different number of subspaces, is computed, and a kernel matrix can be obtained accordingly. Based on the kernel matrix, two feature selection criterions, one supervised and one unsupervised, are proposed to obtain a few representative features in the kernel space. Thus, the curse of dimensionality is alleviated. Experimental results on three multimodal dataset show the proposed feature fusion can outperforms the state-of –the-art by higher accuracy.

Discriminative Graph-Based Fusion of HSI and LiDAR Data for Urban Area Classification

A novel discriminative graph-based fusion (DGF) method is proposed for urban area classification to fuse heterogeneous features from two data sources, i.e., hyperspectral image (HSI) and light detecting and ranging (LiDAR) data. The features include spectral characteristics in HSI, height in LiDAR data, and geometry in image processing technologies like morphological profiles (MPs). Our proposed DGF method couples dimension reduction and heterogeneous feature fusion. The core idea of the proposed method is to search for a projection matrix by minimizing the similarity term that preserves the local geometry of each class and maximizing the dissimilarity term that contains the relation of between-class distance. As a result, the proposed method can pull close together samples of the same class while pushing those of different classes apart in the projected space by fusing graphs constructed by different groups of heterogeneous features. The edges of the graphs are measured by kernel. Furthermore, the multiscale DGF (MS-DGF) is introduced to utilize the capability of similarity measure of different scales of kernel and avoid finding the optimal scale simultaneously. Experiments are conducted on real HSI along with LiDAR data. The corresponding results demonstrate that the proposed method can make an effective fusion of heterogeneous features to make full use of the complementary information of HSI and LiDAR, which facilitates fine classification task of urban area, compared with several state-of-the-art algorithms.

Exploiting score distribution for heterogenous feature fusion in image classification

Over the past decades, features generated by different models have been designed to describe various aspects of object. To connect the complementary information and represent the data properly, effective heterogeneous feature fusion methods are required. Multiple kernel learning (MKL) methods are widely adopted to learn the feature weights and to fuse features on score-level. In this paper, we exploit score distribution to address the feature fusion problem and propose a novel method named score-distribution MKL (SD-MKL) for image classification. Different from existing MKL methods, SD-MKL uses weights which are learned from score curves as a constraint on the weights of kernels. It contains two stages in off-line part: (1) independent data is used to construct reference curves according to classes and feature type; (2) samples and corresponding score-distribution weights are put into multi-kernel support vector machine (MKSVM) to learn feature weights. Our experimental results demonstrate the effect of exploiting score-distribution information on two datasets, which significantly benefits the performance of image classification.

Spatial and semantical label inference for social media

Exploring the spatial and semantical knowledge from messages in social media offers us an opportunity to get a deeper understanding about the mobility and activity of users, which can be leveraged to improve the service quality of online applications like recommender systems. In this paper, we investigate the problem of the spatial and semantical label inference, where the challenges come from three aspects: diverse heterogeneous information, uncertainty of individual mobility, and large-scale sparse data. We address the challenges by exploring two types of data fusion, the fusion of heterogeneous social networks and the fusion of heterogeneous features. We build a 4-dimensional tensor, called spatial---temporal semantical tensor (STST), to model the individual mobility and activity by fusing two heterogeneous social networks, a social media network and a location-based social network (LBSN). To address the challenge arising from diverse heterogeneous information and the uncertainty of individual mobility, we construct three types of heterogeneous features and fuse them with STST by exploring their interdependency relationships. Particularly, a spatial tendency feature is constructed to constrain the inference of individual mobility and reduce the uncertainty. To deal with large-scale sparse data, we propose a parallel contextual tensor factorization (PCTF) to concurrently factorize STST. Finally, we integrate these components into an inference framework, called spatial and semantical label inference SSLI. The results of extensive experiments conducted on real datasets and synthetic datasets verify the effectiveness and efficiency of SSLI.

Reducing uncertainty of dynamic heterogeneous information networks: a fusing reconstructing approach

In real world, a heterogeneous information network (HIN) is often dynamic due to the time varying features of the nodes, and uncertain due to missing values and noise. In this paper, we investigate the problem of reducing the uncertainty of a dynamic HIN, which is an important task for HIN analysis. The challenges are three-fold, the heterogeneity of features, the heterogeneity of constraints, and the dynamic uncertainty. We propose a novel approach, called fusing reconstruction (FRec), which reconstructs the uncertain snapshots of a dynamic HIN in a homogeneous feature space combining two fusions, the fusion of heterogeneous features and the fusion of heterogeneous constraints. To address the challenge of the heterogeneity of features, we propose an invertible fusing transformation (IFT) as the first part of FRec. IFT is a bidirectional transformation, which is able to learn unified latent homogeneous feature representations for heterogeneous nodes and transform them back to the raw heterogeneous feature space by its invertibility. To address the challenge of the heterogeneity of constraints and the challenge of dynamic uncertainty, we propose a heterogeneous constraints fusion based tensor reconstruction model (HCF-TRM) as the second part of FRec. HCF-TRM is able to denoise the uncertain snapshots of a dynamic HIN and recovers the missing values by fusing the spatial smoothness constraint and the temporal smoothness constraint into the tensor reconstruction. At last, the extensive experiments conducted on real datasets and synthetic datasets verify the effectiveness and scalability of FRec.

Jointly Learning Heterogeneous Features for RGB-D Activity Recognition.

In this paper, we focus on heterogeneous features learning for RGB-D activity recognition. We find that features from different channels (RGB, depth) could share some similar hidden structures, and then propose a joint learning model to simultaneously explore the shared and feature-specific components as an instance of heterogeneous multi-task learning. The proposed model formed in a unified framework is capable of: 1) jointly mining a set of subspaces with the same dimensionality to exploit latent shared features across different feature channels, 2) meanwhile, quantifying the shared and feature-specific components of features in the subspaces, and 3) transferring feature-specific intermediate transforms (i-transforms) for learning fusion of heterogeneous features across datasets. To efficiently train the joint model, a three-step iterative optimization algorithm is proposed, followed by a simple inference model. Extensive experimental results on four activity datasets have demonstrated the efficacy of the proposed method. A new RGB-D activity dataset focusing on human-object interaction is further contributed, which presents more challenges for RGB-D activity benchmarking.

A ℓ2, 1 norm regularized multi-kernel learning for false positive reduction in Lung nodule CAD

Objective: The aim of this paper is to describe a novel algorithm for False Positive Reduction in lung nodule Computer Aided Detection(CAD).Methods: In this paper, we describes a new CT lung CAD method which aims to detect solid nodules. Specially, we proposed a multi-kernel classifier with a ℓ2, 1 norm regularizer for heterogeneous feature fusion and selection from the feature subset level, and designed two efficient strategies to optimize the parameters of kernel weights in non-smooth ℓ2, 1 regularized multiple kernel learning algorithm. The first optimization algorithm adapts a proximal gradient method for solving the ℓ2, 1 norm of kernel weights, and use an accelerated method based on FISTA; the second one employs an iterative scheme based on an approximate gradient descent method.Results: The results demonstrates that the FISTA-style accelerated proximal descent method is efficient for the ℓ2, 1 norm formulation of multiple kernel learning with the theoretical guarantee of the convergence rate. Moreover, the experimental results demonstrate the effectiveness of the proposed methods in terms of Geometric mean (G-mean) and Area under the ROC curve (AUC), and significantly outperforms the competing methods.Conclusions: The proposed approach exhibits some remarkable advantages both in heterogeneous feature subsets fusion and classification phases. Compared with the fusion strategies of feature-level and decision level, the proposed ℓ2, 1 norm multi-kernel learning algorithm is able to accurately fuse the complementary and heterogeneous feature sets, and automatically prune the irrelevant and redundant feature subsets to form a more discriminative feature set, leading a promising classification performance. Moreover, the proposed algorithm consistently outperforms the comparable classification approaches in the literature.

A multi-kernel based framework for heterogeneous feature selection and over-sampling for computer-aided detection of pulmonary nodules

Classification plays a critical role in False Positive Reduction (FPR) in lung nodule Computer Aided Detection (CAD). To achieve effective recognition of nodule, many machine learning methods have been proposed. However, multiple heterogeneous feature subsets, high dimensional irrelevant features, as well as imbalanced distribution between the nodule and non-nodule classes typically makes this problem challenging. To solve these challenges, we proposed a multi-kernel based framework for feature selection and imbalanced data learning in Lung nodule CAD, involving multiple kernel learning with a ℓ2,1 norm regularizer for heterogeneous feature fusion and selection from the feature subset level, a multi-kernel feature selection based on pairwise similarities from the feature level, and a multi-kernel over-sampling for the imbalanced data learning. Experimental results demonstrate the effectiveness of the proposed method in terms of Geometric mean (G-mean) and Area under the ROC curve (AUC), and consistently outperform the competing methods.

Hand posture recognition based on heterogeneous features fusion of multiple kernels learning

As a rapid developing research topic in the machine vision field, image-based hand posture recognition has the potential to be an efficient and intuitive tool of human-computer interaction. For improving the accuracy of multi-class hand postures and extending the algorithm generalization,a novel hand posture recognition method is proposed by integrating the multiple image features and multiple kernels learning support vector machine(SVM). Firstly, three types of feature descriptors are extracted to describe the characteristics of a hand posture image. Shape context descriptor represents distribution characteristics of the edge points of the hand posture image. Pyramid histogram of oriented gradient describes characteristics of local and global shape effectively. The Bag of Feature(BOF) algorithm describes the surface texture characteristics of the posture image. Secondly, the Chamfer kernel and histogram intersection kernel are rebuilt to obtain the basis kernels of the features. And the combined kernel is constructed by weighting the basis kernels.So the heterogeneous features fusion realizes. Finally, the classification model and optimal fusion weights are calculated by using multiple kernels learning algorithm. The unknown category posture can be recognized by the trained multiple kernels of SVM. Experiments on Jochen Triesch's hand posture dataset demonstrate that the proposed method obtains higher recognition rate than the traditional single-kernel classifier and other recent methods.

Online primal–dual learning for a data-dependent multi-kernel combination model with multiclass visual categorization applications

The task of visual categorization becomes very challenging as the number of samples or classes increases. This is mainly due to large memory requirements or a crowded semantic space, which causes class separation difficulties. In this paper, a new approach called OPrDuM2 (online primal–dual multiple kernel multiple class) is proposed. This new algorithm can significantly reduce the complexity of visual categorization. Under this approach, an elegant heterogeneous feature fusion machine is used to consolidate complementary class-discriminative information, and online learning is applied to deal with large sample sizes. To be specific, a data-dependent multi-kernel machine that fuses multiple heterogeneous features in a nonlinear manner is defined by extending the multi-kernel learning model, and multivariate hinge loss and conservative updating rules are used to increase the sample-level sparsity in the model. Given the intrinsic hierarchical structure of samples, features, and classes, an innovative triple mixed-norm regularizer with strong convexity is utilized to facilitate optimization. An iterative solution is derived using the language of duality, leading to a low regret bound of OT, where T is the number of online iterations. This paper details a proof of the algorithm’s convergence, and describes extensive visual data classification experiments that demonstrate the effectiveness and robustness of this new approach.

RGB-D Hand-Held Object Recognition Based on Heterogeneous Feature Fusion

Object recognition has many applications in human-machine interaction and multimedia retrieval. However, due to large intra-class variability and inter-class similarity, accurate recognition relying only on RGB data is still a big challenge. Recently, with the emergence of inexpensive RGB-D devices, this challenge can be better addressed by leveraging additional depth information. A very special yet important case of object recognition is hand-held object recognition, as manipulating objects with hands is common and intuitive in human-human and human-machine interactions. In this paper, we study this problem and introduce an effective framework to address it. This framework first detects and segments the hand-held object by exploiting skeleton information combined with depth information. In the object recognition stage, this work exploits heterogeneous features extracted from different modalities and fuses them to improve the recognition accuracy. In particular, we incorporate handcrafted and deep learned features and study several multi-step fusion variants. Experimental evaluations validate the effectiveness of the proposed method.

Salient environmental sound detection framework for machine awareness

Auditory perception is an essential part of environment perception, in which the saliency detection is not only the fundamental basis but also an efficient way of achieving this task. For artificial machines, intelligent perception approach of sound is required to provide awareness as the initiatory step of artificial consciousness. In this paper, a novel salient environment sound detection framework for machine awareness is proposed. The framework is based on the heterogeneous saliency features from both image and acoustic channels. To improve the efficiency of proposed framework, (1) a global informative saliency estimation approach is initially proposed based on short-term Shannon entropy; (2) a series of auditory saliency detection methods is presented to obtain the spectral and temporal saliency features from power spectral density and mel-frequency cepstral coefficients, respectively; (3) a computational bio-inspired inhibition of return model is proposed for saliency verification to improve the accuracy of detection; (4) a heterogeneous saliency feature fusion approach is introduced to form the final auditory saliency map by combining the acoustic and image saliency features together. Environmental sounds which collected from real world are applied to verify the superiority of the proposed framework. The results show that, the proposed framework is more effective for the detection of the overlapped salient sounds, and is more robust to the background noise compared with the conventional approach.

PredPPCrys: accurate prediction of sequence cloning, protein production, purification and crystallization propensity from protein sequences using multi-step heterogeneous feature fusion and selection.

X-ray crystallography is the primary approach to solve the three-dimensional structure of a protein. However, a major bottleneck of this method is the failure of multi-step experimental procedures to yield diffraction-quality crystals, including sequence cloning, protein material production, purification, crystallization and ultimately, structural determination. Accordingly, prediction of the propensity of a protein to successfully undergo these experimental procedures based on the protein sequence may help narrow down laborious experimental efforts and facilitate target selection. A number of bioinformatics methods based on protein sequence information have been developed for this purpose. However, our knowledge on the important determinants of propensity for a protein sequence to produce high diffraction-quality crystals remains largely incomplete. In practice, most of the existing methods display poorer performance when evaluated on larger and updated datasets. To address this problem, we constructed an up-to-date dataset as the benchmark, and subsequently developed a new approach termed ‘PredPPCrys’ using the support vector machine (SVM). Using a comprehensive set of multifaceted sequence-derived features in combination with a novel multi-step feature selection strategy, we identified and characterized the relative importance and contribution of each feature type to the prediction performance of five individual experimental steps required for successful crystallization. The resulting optimal candidate features were used as inputs to build the first-level SVM predictor (PredPPCrys I). Next, prediction outputs of PredPPCrys I were used as the input to build second-level SVM classifiers (PredPPCrys II), which led to significantly enhanced prediction performance. Benchmarking experiments indicated that our PredPPCrys method outperforms most existing procedures on both up-to-date and previous datasets. In addition, the predicted crystallization targets of currently non-crystallizable proteins were provided as compendium data, which are anticipated to facilitate target selection and design for the worldwide structural genomics consortium. PredPPCrys is freely available at http://www.structbioinfor.org/PredPPCrys.

Online heterogeneous feature fusion machines for visual recognition

Heterogeneous Feature Fusion Machines (HFFM) is a kernel based logistic regression model that effectively fuses multiple features for visual recognition tasks. However, the batch mode solution for HFFM, ‘Block Coordinate Gradient Descent’ (BCGD) has the same low efficiency and poor scalability as the most batch algorithms do. In this paper, we describe a newly developed online learning algorithm in multiple Reproducing Kernel Hilbert Spaces for solving HFFM model. This new algorithm is called OLHFFM, i.e. Online HFFM. OLHFFM is novel combination of kernel-based learning technique with dual averaging gradient descent methods. In addition, group LASSO regularization technique is used in OLHFFM for finding important explanatory coefficients that are related to support samples in group manner. The effectiveness of OLHFFM has been demonstrated by a number of experiments that were conducted on public event, object dataset, as well as on large scale handwritten digital dataset. Using the OLHFFM approach, we have achieved almost equivalent recognition performance to that using batch-mode approach. Experiments conducted on both MIT Caltech-6 and challenging VOC2011 TrainVal object datasets show that OLHFFM is superior in performance to kernel based online learning approaches such as ILK or NORMA. In addition, the classification performance of OLHFFM approach as demonstrated by the experiments conducted on large scale MNIST dataset is comparable to or better than that of the current state-of-the-art online multiple kernel learning approaches such as OM-2, UFO-MKL, OMCL and OMKL. Extensive experiments on visual data classification demonstrate the effectiveness and robustness of the new OLHFFM approach.