Feature Space Representation Research Articles

Feature Space Exploration For Planning Initial Benthic AUV Surveys

Special-purpose Autonomous Underwater Vehicles (AUVs) are utilized for benthic (seafloor) surveys, where the vehicle collects optical imagery of the seafloor. Due to the small-sensor footprint of the cameras and the vast areas to be surveyed, these AUVs can not feasibly collect full coverage imagery of areas larger than a few tens of thousands of square meters. Therefore it is necessary for AUV paths to sample the surveys areas sparsely, yet effectively. Broad-scale acoustic bathymetric data are readily available over large areas and are often a useful prior of seafloor cover. As such, prior bathymetry can be used to guide AUV data collection. This research proposes methods for planning initial AUV surveys that efficiently explore a feature space representation of the bathymetry, in order to sample from a diverse set of bathymetric terrain. This will enable the AUV to visit areas that likely contain unique habitats and are representative of the entire survey site. We propose several information gathering planners that utilize a feature space exploration reward, to plan freeform paths or to optimize the placement of a survey template. The suitability of these methods to plan AUV surveys is evaluated based on the coverage of the feature space and also the ability to visit all classes of benthic habitat on the initial dive. Informative planners based on Rapidly expanding Random Trees (RRT) and Monte Carlo Tree Search (MCTS) were found to be the most effective. This is a valuable tool for AUV surveys as it increases the utility of initial dives. It also delivers a comprehensive training set to learn the relationship between acoustic bathymetry and visually derived seafloor classifications.

CAMR: cross-aligned multimodal representation learning for cancer survival prediction.

Accurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances in survival prediction methods suggest that integrating complementary information from different modalities, e.g. histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause the so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning and therefore pose a great challenge to further improvements of cancer survival prediction. To solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data. CAMR is freely available at https://github.com/wxq-ustc/CAMR. Supplementary data are available at Bioinformatics online.

EMSCNet: Efficient Multisample Contrastive Network for Remote Sensing Image Scene Classification

Significant progress has been achieved in remote sensing image scene classification (RSISC) with the development of convolutional neural networks (CNNs) and vision transformers (ViT). However, high intra-class diversity and inter-class similarity are still enormous challenges for RSISC. Metric learning can effectively improve the discriminative ability of deep representations by constraining the distance between features. Previous metric learning methods only optimize the feature space representation through metric function, ignoring the information interaction between samples. For complex scene images, similarity and discriminative knowledge need to be summarized from the multiple positive and negative pairs. We propose a novel efficient multi-sample contrastive network (EMSCNet) to integrate knowledge from multiple samples. Specifically, we construct a dynamic dictionary with momentum updates to mine positive and negative pairs from the entire dataset. Then, the similarity and discriminative knowledge between samples are summarized by introducing a contrastive module. Finally, the knowledge of the contrastive module is transferred to the backbone classifier through knowledge distillation. The proposed contrastive module can be easily embedded into the training process of CNNs or ViT and removed during inference. Experimental results conducted on three datasets demonstrate the effectiveness of the proposed approach.

Toward robust process monitoring of complex process industries based on denoising sparse auto-encoder

Robust industrial process monitoring is crucial to ensure production safety and product quality stability. However, the process monitoring of high-dimensional, nonlinear, complex industrial processes is still challenging due to their inherent complexities, such as multi-phase, multi-field, and tight coupling of multiple sub-processes. In this article, a simple yet robust nonlinear process monitoring scheme based on a kind of denoising sparse auto-encoder (DSAE) is proposed. Specifically, a novel hybrid auto-encoder, namely DSAE, is established, to address the strong redundancy, nonlinearity and noise interference in process variables by integrating a sparse auto-encoder (SAE) and a denoising auto-encoder (DAE). Successively, an online process monitoring model is established by introducing two new process monitoring statistics computed on the DSAE-based feature representation space and residual space, respectively. Moreover, the Kernel density estimation (KDE) approach is adopted to determine the corresponding control limits of the monitoring statistics, which can avoid the conventional empirical assumption of F or Chi-square distribution on monitoring statistics. Extensive confirmative and comparative experiments conducted for the fault monitoring of a nonlinear numerical simulation system, the benchmark Tennessee Eastman process (TEP) and ventilation exhaust fans (VEFs) in the continuous casting process (CCP) from a top steel plant in China show that the proposed method is promising. Specifically, the proposed method performs favorably against the representative process monitoring methods, and it has especially a stronger robustness against noise and uncertain interference in nonlinear process systems.

Context-dependent emotion recognition

Most previous methods for emotion recognition focus on facial emotion and ignore the rich context information that implies important emotion states. To make full use of the contextual information to make up for the facial information, we propose the Context-Dependent Net (CD-Net) for robust context-aware human emotion recognition. Inspired by the long-range dependency of the transformer, we introduce the tubal transformer which forms the shared feature representation space to facilitate the interactions among the face, body, and context features. Besides, we introduce the hierarchical feature fusion to recombine the enhanced multi-scale face, body, and context features for emotion classification. Experimentally, we verify the effectiveness of the proposed CD-Net on the two large emotion datasets, CAER-S and EMOTIC. On the one hand, the quantitative evaluation results demonstrate the superiority of the proposed CD-Net over other state-of-the-art methods. On the other hand, the visualization results show CD-Net can capture the dependencies among the face, body, and context components and focus on the important features related to the emotion.

Unsupervised feature selection guided by orthogonal representation of feature space

Feature selection has been an outstanding strategy in eliminating redundant and inefficient features in high-dimensional data. This paper introduces a novel unsupervised feature selection based on the matrix factorization, namely Unsupervised Feature Selection Guided by Orthogonal Representation (UFGOR). The orthogonality between a pair of variables refers to a specific case of linear independence such that they are perfectly uncorrelated. Motivated by the benefits of the orthogonality concept, the proposed UFGOR method is established based on the distance between the selected feature set and an orthogonal set corresponding to the whole feature space. Moreover, this orthogonal set is generated via QR-matrix factorization over the whole features and is employed as the compact representation of data matrix. In the next step, an unsupervised feature selection method is performed through the matrix factorization of the generated orthogonal set. Additionally, a dual-correlation model is utilized in the objective function of UFGOR to simultaneously consider both the local correlation in a set of selected features and the global correlation among the samples of a data. A detailed convergence analysis in line with an effective iterative algorithm proposed for the UFGOR method is also given. Numerical experiments on several real-world datasets illustrate the superior efficiency of our approach in comparison with some state-of-the-art unsupervised feature selection methods.

Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement

This paper proposes a novel multimodal self-supervised architecture for energy-efficient audio-visual (AV) speech enhancement that integrates Graph Neural Networks with canonical correlation analysis (CCA-GNN). The proposed approach lays its foundations on a state-of-the-art CCA-GNN that learns representative embeddings by maximizing the correlation between pairs of augmented views of the same input while decorrelating disconnected features. The key idea of the conventional CCA-GNN involves discarding augmentation-variant information and preserving augmentation-invariant information while preventing capturing of redundant information. Our proposed AV CCA-GNN model deals with multimodal representation learning context. Specifically, our model improves contextual AV speech processing by maximizing canonical correlation from augmented views of the same channel and canonical correlation from audio and visual embeddings. In addition, it proposes a positional node encoding that considers a prior-frame sequence distance instead of a feature-space representation when computing the node’s nearest neighbors, introducing temporal information in the embeddings through the neighborhood’s connectivity. Experiments conducted on the benchmark ChiME3 dataset show that our proposed prior frame-based AV CCA-GNN ensures a better feature learning in the temporal context, leading to more energy-efficient speech reconstruction than state-of-the-art CCA-GNN and multilayer perceptron.

Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation

Language resources are the main factor in speech-emotion-recognition (SER)-based deep learning models. Thai is a low-resource language that has a smaller data size than high-resource languages such as German. This paper describes the framework of using a pretrained-model-based front-end and back-end network to adapt feature spaces from the speech recognition domain to the speech emotion classification domain. It consists of two parts: a speech recognition front-end network and a speech emotion recognition back-end network. For speech recognition, Wav2Vec2 is the state-of-the-art for high-resource languages, while XLSR is used for low-resource languages. Wav2Vec2 and XLSR have proposed generalized end-to-end learning for speech understanding based on the speech recognition domain as feature space representations from feature encoding. This is one reason why our front-end network was selected as Wav2Vec2 and XLSR for the pretrained model. The pre-trained Wav2Vec2 and XLSR are used for front-end networks and fine-tuned for specific languages using the Common Voice 7.0 dataset. Then, feature vectors of the front-end network are input for back-end networks; this includes convolution time reduction (CTR) and linear mean encoding transformation (LMET). Experiments using two different datasets show that our proposed framework can outperform the baselines in terms of unweighted and weighted accuracies.

An asymmetrical-structure auto-encoder for unsupervised representation learning of skeleton sequences

In this paper, we propose a novel framework for unsupervised representation learning using a structure-asymmetrical auto-encoder in which a 2D-CNN-based encoder learns separable spatiotemporal representations in a low-dimensional feature space under the supervision of salient skeleton motion cues. This study addresses the problem of learning action representations of skeleton sequences. The network captures not only correlations of adjacent joints but also long-term motion dependencies by using the proposed unsupervised training, which leads to the advantage that similar movements are gathered around the same cluster, whereas different movements are gathered around distinct clusters. Our method is unsupervised and does not rely on annotations to associate skeleton sequences with actions. Experimental results clearly showed the effectiveness of the proposed representation learning, and improvements compared with skeleton-based generative learning methods. When the proposed network was fine-tuned with partial labeled data, our results still outperformed some fully supervised methods.

Cross-modal distillation for RGB-depth person re-identification

Person re-identification is a key challenge for surveillance across multiple sensors. Prompted by the advent of powerful deep learning models for visual recognition, and inexpensive RGB-D cameras and sensor-rich mobile robotic platforms, e.g. self-driving vehicles, we investigate the relatively unexplored problem of cross-modal re-identification of persons between RGB (color) and depth images. The considerable divergence in data distributions across different sensor modalities introduces additional challenges to the typical difficulties like distinct viewpoints, occlusions, and pose and illumination variation. While some work has investigated re-identification across RGB and infrared, we take inspiration from successes in transfer learning from RGB to depth in object detection tasks. Our main contribution is a novel method for cross-modal distillation for robust person re-identification, which learns a shared feature representation space of person’s appearance in both RGB and depth images. In addition, we propose a cross-modal attention mechanism where the gating signal from one modality can dynamically activate the most discriminant CNN filters of the other modality. The proposed distillation method is compared to conventional and deep learning approaches proposed for other cross-domain re-identification tasks. Results obtained on the public BIWI and RobotPKU datasets indicate that the proposed method can significantly outperform the state-of-the-art approaches by up to 16.1% in mean Average Precision (mAP), demonstrating the benefit of the distillation paradigm. The experimental results also indicate that using cross-modal attention allows to improve recognition accuracy considerably with respect to the proposed distillation method and relevant state-of-the-art approaches.11Code: https://github.com/frhf/cross-modal-distillation-reidentification.

SCL-Net: An End-to-End Supervised Contrastive Learning Network for Hyperspectral Image Classification

In recent years, deep learning presents a promising performance in hyperspectral image (HSI) classification, due to the powerful capability of automatically learning deep semantic characteristics of images. However, it is still difficult to learn highly discriminative features when limited samples are available for training a deep network. Focused on this issue, a novel end-to-end supervised contrastive learning network (SCL-Net) for spectral-spatial classification is proposed, in this paper. Instead of learning features of the individual sample, the supervised contrastive learning is introduced to capture the similarity and dissimilarity distribution properties of sample pairs in feature representation space. In this way, the need for plenty of training samples will be alleviated while an effective network training mechanism is provided for learning highly separative features. Here, the SCL-Net mainly consists of one pair-wise contrastive learning (PCL) sub-network and one multi-level spectral-spatial information fusion (MLSIF) sub-network. For the PCL sub-network, spectral vectors are projected into deep spectral features based on convolutional operators, which are then followed by distance evaluation between “positive” pairs of similar samples and “negative” pairs of dissimilar ones. Then, a spectral distance matrix is constructed to push the network to gradually learn better features of higher intra-class compactness and inter-class dispersion. For the MLSIF sub-network, a hybrid feature-decision fusion strategy is designed, where spatial and spectral features are jointly exploited to further boost classification performance. In specific, the feature fusion is conducted by connecting low/mid/high-level spectral and spatial features via weighting, while multiple class estimations based on multi-level fusion features are adaptively integrated via probabilistic decision fusion. Overall, these two sub-networks are collaboratively trained in one framework, by optimizing a defined joint loss function consisting of a contrastive loss and a cross-entropy loss. Compared with several state-of-the-art methods, the proposed method yields a superior classification performance in terms of both objective metrics and visual performance.

The Recurrent Reinforcement Learning Crypto Agent

We demonstrate a novel application of online transfer learning for a digital assets trading agent. This agent uses a powerful feature space representation in the form of an echo state network, the output of which is made available to a direct, recurrent reinforcement learning agent. The agent learns to trade the XBTUSD (Bitcoin versus US Dollars) perpetual swap derivatives contract on BitMEX on an intraday basis. By learning from the multiple sources of impact on the quadratic risk-adjusted utility that it seeks to maximise, the agent avoids excessive over-trading, captures a funding profit, and can predict the market's direction. Overall, our crypto agent realises a total return of 350\%, net of transaction costs, over roughly five years, 71\% of which is down to funding profit. The annualised information ratio that it achieves is 1.46.

Nessy: A Neuro-Symbolic System for Label Noise Reduction

Noisy labels represent one of the key issues in supervised machine learning. Existing work for label noise reduction mainly takes a probabilistic approach that infers true labels from data distributions in low-level feature spaces. Such an approach is not only limited by its capability to learn high-quality data representations, but also by the low predictive power of data distributions in inferring true classes. To address those problems, we introduce Nessy, a neuro-symbolic system that integrates deep probabilistic modeling and symbolic knowledge for label noise reduction. Our deep probabilistic model infers the true classes of data instances with noisy labels by exploiting data distributions in an underlying latent feature representation space. For data instances where inference is not reliable enough, Nessy extracts symbolic rules and ranks them according to several utility metrics. Top-ranking rules are injected into the deep probabilistic model via expectation regularization, i.e., via a posterior regularization term constraining the class distribution in the objective function. In a real deployment over multiple relation extraction tasks, we demonstrate that Nessy is able to significantly improve the state of the art, by 7% accuracy and 10.7% AUC on average.

Hierarchical GAN-Tree and Bi-Directional Capsules for multi-label image classification

Compared with the flat multi-label image classification, the hierarchical structure reserves a richer source of structural information to represent complicated relationships between labels in the real world. However, existing multi-label image classification methods focus on the accuracy of label prediction, ignoring the structural information embedded in the hierarchical label space. Furthermore, they hardly form the relevant visual feature space corresponding to the hierarchical label structure. In this paper, we propose a novel hierarchical framework based on the feature and label structural information named Hierarchical GAN-Tree and Bi-Directional Capsules (HGT&BC) to address these problems. We conduct Hierarchical GAN-Tree for feature space representation and Hierarchical Bi-Directional Capsules for label space classification, respectively. Hierarchical GAN-Tree generates hierarchical feature space using the unsupervised divisive clustering pattern according to the hierarchical structure, alleviating the mode-collapse of generators and the overfitting manifestation of conventional GANs. Hierarchical Bi-Directional Capsules utilize the hierarchical label structure in iterations of top-down and bottom-up processes: the top-down process integrates hierarchical relationships into the probability computation to enhance partial hierarchical relationships; the bottom-up process modifies the dynamic routing mechanism between capsules to represent semantic objects for the comprehensive global hierarchical classifiers. Owing to the two components, HGT&BC successfully expresses the hierarchical relationships in both feature and label space and improves the performance of multi-label image classification. Extensive experimental results on four benchmark datasets demonstrate the effectiveness and efficiency of our hierarchical framework in practice.

Subspace alignment based on an extreme learning machine for electronic nose drift compensation

The drift caused by gas sensors has always been a bottleneck in the development of electronic nose (E-nose) systems. Traditional drift compensation methods directly correct the drift components, making such approaches time-consuming and laborious. In the field of E-nose drift compensation, cross-domain adaption learning is an efficient technique. In this paper, we propose a novel subspace alignment extreme learning machine (SAELM) that considers multiple criteria to construct a unified extreme learning machine (ELM)-based feature representation space and thus achieve domain alignment. First, the method minimizes both the geometric and statistical distributions between different domains. Second, the dependence between features and labels is enhanced using the Hilbert–Schmidt independence criterion (HSIC) to alleviate the blurring of the correspondence between the two caused by drift. Third, to improve the feature extraction ability of the subspace learning method, the l2,1 norm is leveraged to constrain the output weights of the ELM. The aim of this method is to learn a robust subspace to increase the consistency between domains and enhance the feature–label​ dependency of the source domain while preserving the intrinsic information of both domains. Extensive experiments on sensor drift data are conducted, and the proposed SAELM method yields the greatest improvements on E-nose drift datasets.

Dual-Constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior

Nonnegativity based matrix factorization is usually powerful for learning the parts-based “shallow” representation, however it fails to discover deep hidden information within both the basis concept and representation spaces. In this paper, we therefore propose a new dual-constrained deep semi-supervised coupled factorization network (DS2CF-Net) for learning hierarchical representations. DS2CF-Net is formulated as the joint partial-label and structure-constrained deep factorization network using multi-layers of linear transformations, which coupled updates the basic concepts and new representations in each layer. An error correction mechanism with feature fusion strategy is also integrated between consecutive layers to improve the representation ability of features. To improve the discriminating abilities of both representation and coefficients in feature space, we clearly consider how to enrich the prior knowledge by the coefficients-based label prediction, and incorporate the enriched prior knowledge as the additional label and structure constraints. To be specific, the label constraint enables the intra-class samples to have the same coordinate in the feature space, while the structure constraint forces the coefficients in each layer to be block-diagonal so that the enriched prior knowledge are more accurate. Besides, we integrate the adaptive dual-graph learning to retain the locality structures of both the data manifold and feature manifold in each layer. Finally, a fine-tuning process is performed to refine the structure-constrained matrix and data weight matrix in each layer using the predicted labels for more accurate representations. Extensive simulations on public databases show that our method can obtain state-of-the-art performance.

Multiscale Reweighted Stochastic Embedding: Deep Learning of Collective Variables for Enhanced Sampling.

Machine learning methods provide a general framework for automatically finding and representing the essential characteristics of simulation data. This task is particularly crucial in enhanced sampling simulations. There we seek a few generalized degrees of freedom, referred to as collective variables (CVs), to represent and drive the sampling of the free energy landscape. In theory, these CVs should separate different metastable states and correspond to the slow degrees of freedom of the studied physical process. To this aim, we propose a new method that we call multiscale reweighted stochastic embedding (MRSE). Our work builds upon a parametric version of stochastic neighbor embedding. The technique automatically learns CVs that map a high-dimensional feature space to a low-dimensional latent space via a deep neural network. We introduce several new advancements to stochastic neighbor embedding methods that make MRSE especially suitable for enhanced sampling simulations: (1) weight-tempered random sampling as a landmark selection scheme to obtain training data sets that strike a balance between equilibrium representation and capturing important metastable states lying higher in free energy; (2) a multiscale representation of the high-dimensional feature space via a Gaussian mixture probability model; and (3) a reweighting procedure to account for training data from a biased probability distribution. We show that MRSE constructs low-dimensional CVs that can correctly characterize the different metastable states in three model systems: the Müller-Brown potential, alanine dipeptide, and alanine tetrapeptide.

HDNet: Hybrid Distance Network for semantic segmentation

Semantic segmentation is currently solved as a pixel-wise labeling task, which predicts the label of each pixel based on its features. However, current methods isolate the relations of points in a feature map and cause the discontinuous segmentation results. In order to solve this problem, we propose a Hybrid Distance Network to measure the distance from two aspects. First, the Hybrid Distance Relation is proposed to model the relations between a point and its context regions to capture contexts in a feature map by an elegant combination of positional distance and high-dimension feature distance. Then, a Location Aware Attention module is proposed to efficiently sample the contexts by the positional distance and produces sparse Hybrid Distance Relations. It synthesizes the different contexts of each point and generates position-wise attention value to compact object-level representation. During the training step, High-dimension Feature Distance loss is also presented as an auxiliary loss to compact category-level representation in feature space. Experiments show that the proposed HDNet achieves state-of-the-art performance with interpretability and efficiency on three challenging semantic segmentation benchmarks: Pascal Context, ADE20K, and COCO Stuff 10 K.

Learning feature representation of Iberian ceramics with automatic classification models

Abstract In Cultural Heritage inquiries, a common requirement is to establish time-based trends between archaeological artifacts belonging to different periods of a given culture, enabling among other things to determine chronological inferences with higher accuracy and precision. Among these, pottery vessels are significantly useful, given their relative abundance in most archaeological sites. However, this very abundance makes difficult and complex an accurate representation, since no two of these vessels are identical, and therefore classification criteria must be justified and applied. For this purpose, we propose the use of deep learning architectures to extract automatically learned features without prior knowledge or engineered features. By means of transfer learning, we retrained a Residual Neural Network with a binary image database of Iberian wheel-made pottery vessels’ profiles. These vessels pertain to archaeological sites located in the upper valley of the Guadalquivir River (Spain). The resulting model can provide an accurate feature representation space, which can automatically classify profile images, achieving a mean accuracy of 0.96 with an f -measure of 0.96 . This accuracy is remarkably higher than other state-of-the-art machine learning approaches, where several feature extraction techniques were applied together with multiple classifier models. These results provide novel strategies to current research in automatic feature representation and classification of different objects of study within the Archaeology domain.

Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AI

AI is remarkably successful and outperforms human experts in certain tasks, even in complex domains such as medicine. Humans on the other hand are experts at multi-modal thinking and can embed new inputs almost instantly into a conceptual knowledge space shaped by experience. In many fields the aim is to build systems capable of explaining themselves, engaging in interactive what-if questions. Such questions, called counterfactuals, are becoming important in the rising field of explainable AI (xAI). Our central hypothesis is that using conceptual knowledge as a guiding model of reality will help to train more explainable, more robust and less biased machine learning models, ideally able to learn from fewer data. One important aspect in the medical domain is that various modalities contribute to one single result. Our main question is “How can we construct a multi-modal feature representation space (spanning images, text, genomics data) using knowledge bases as an initial connector for the development of novel explanation interface techniques?”. In this paper we argue for using Graph Neural Networks as a method-of-choice, enabling information fusion for multi-modal causability (causability – not to confuse with causality – is the measurable extent to which an explanation to a human expert achieves a specified level of causal understanding). The aim of this paper is to motivate the international xAI community to further work into the fields of multi-modal embeddings and interactive explainability, to lay the foundations for effective future human–AI interfaces. We emphasize that Graph Neural Networks play a major role for multi-modal causability, since causal links between features can be defined directly using graph structures.