Learning High-level Representations Research Articles

Generalizing Graph Neural Networks on Out-of-Distribution Graphs.

Graph Neural Networks (GNNs) are proposed without considering the agnostic distribution shifts between training graphs and testing graphs, inducing the degeneration of the generalization ability of GNNs in Out-Of-Distribution (OOD) settings. The fundamental reason for such degeneration is that most GNNs are developed based on the I.I.D hypothesis. In such a setting, GNNs tend to exploit subtle statistical correlations existing in the training set for predictions, even though it is a spurious correlation. This learning mechanism inherits from the common characteristics of machine learning approaches. However, such spurious correlations may change in the wild testing environments, leading to the failure of GNNs. Therefore, eliminating the impact of spurious correlations is crucial for stable GNN models. To this end, in this paper, we argue that the spurious correlation exists among subgraph-level units and analyze the degeneration of GNN in causal view. Based on the causal view analysis, we propose a general causal representation framework for stable GNN, called StableGNN. The main idea of this framework is to extract high-level representations from raw graph data first and resort to the distinguishing ability of causal inference to help the model get rid of spurious correlations. Particularly, to extract meaningful high-level representations, we exploit a differentiable graph pooling layer to extract subgraph-based representations by an end-to-end manner. Furthermore, inspired by the confounder balancing techniques from causal inference, based on the learned high-level representations, we propose a causal variable distinguishing regularizer to correct the biased training distribution by learning a set of sample weights. Hence, GNNs would concentrate more on the true connection between discriminative substructures and labels. Extensive experiments are conducted on both synthetic datasets with various distribution shift degrees and eight real-world OOD graph datasets. The results well verify that the proposed model StableGNN not only outperforms the state-of-the-arts but also provides a flexible framework to enhance existing GNNs. In addition, the interpretability experiments validate that StableGNN could leverage causal structures for predictions.

A novel intelligent fault diagnosis method of bearing based on multi-head self-attention convolutional neural network

Abstract Deep learning (DL) has been widely used in bearing fault diagnosis. In particular, convolutional neural networks (CNNs) improve diagnosis accuracy by extracting excellent fault features. However, CNN lacks an explicit learning mechanism to distinguish between different fault characteristics in the input signal to the diagnosis results. This article presents a new end-to-end depth framework called multi-head self-attention convolution neural network (MSA-CNN) for bearing fault diagnosis. Firstly, we adopt a data pre-processing method that directly converts one-dimensional (1D) original signals into two-dimensional (2D) grayscale images, which is simple to implement and preserves the complete information of the original signal. Secondly, multi-head self-attention (MSA) is first constructed to aggregate the global information and adaptively assign weights to the input signal's features. Thirdly, the CNN with small-scale kernels extracted detailed local features. Finally, the learned high-level representations are fed into the full connect (FC) layer for fault diagnosis. The performance of the MSA-CNN is validated on different datasets. The results show that the proposed MSA-CNN can significantly improve fault diagnosis accuracy compared with the other state-of-the-art methods and has excellent noise immunity performance.

Diversifying Collaborative Filtering via Graph Spreading Network and Selective Sampling.

Graph neural network (GNN) is a robust model for processing non-Euclidean data, such as graphs, by extracting structural information and learning high-level representations. GNN has achieved state-of-the-art recommendation performance on collaborative filtering (CF) for accuracy. Nevertheless, the diversity of the recommendations has not received good attention. Existing work using GNN for recommendation suffers from the accuracy-diversity dilemma, where slightly increases diversity while accuracy drops significantly. Furthermore, GNN-based recommendation models lack the flexibility to adapt to different scenarios' demands concerning the accuracy-diversity ratio of their recommendation lists. In this work, we endeavor to address the above problems from the perspective of aggregate diversity, which modifies the propagation rule and develops a new sampling strategy. We propose graph spreading network (GSN), a novel model that leverages only neighborhood aggregation for CF. Specifically, GSN learns user and item embeddings by propagating them over the graph structure, utilizing both diversity-oriented and accuracy-oriented aggregations. The final representations are obtained by taking the weighted sum of the embeddings learned at all layers. We also present a new sampling strategy that selects potentially accurate and diverse items as negative samples to assist model training. GSN effectively addresses the accuracy-diversity dilemma and achieves improved diversity while maintaining accuracy with the help of a selective sampler. Moreover, a hyper-parameter in GSN allows for adjustment of the accuracy-diversity ratio of recommendation lists to satisfy the diverse demands. Compared to the state-of-the-art model, GSN improved R @20 by 1.62% , N @20 by 0.67% , G @20 by 3.59% , and E @20 by 4.15% on average over three real-world datasets, verifying the effectiveness of our proposed model in diversifying overall collaborative recommendations.

A novel two-stream multi-head self-attention convolutional neural network for bearing fault diagnosis

Bearing is one of the most prone to failure components, and the most commonly used supporting parts in rotating machinery. There are important economic and safety significance for studying effective bearing fault diagnosis method. Deep learning has been widely applied to intelligent fault diagnosis of rotating machines. In particular, convolutional neural network (CNN) improves diagnostic accuracy due to automatic feature extraction ability. However, CNN cannot fully use the temporal information of the input signals and extract global features. This article presents a new depth model called two-stream multi-head self-attention convolutional neural network (TSMSCNN) for bearing fault diagnosis to solve the above problems. First, we obtain temporal information through the difference of data points in the input signal. Next, the temporal information and original signal are used as input for two CNN branches. Then, the multi-head self-attention (MS) mechanism adaptively aggregates the features and assigns weights to the features extracted by CNN. Finally, the learned high-level representations are fed into the full connect (FC) layer for fault diagnosis. Experimental results prove that the proposed TSMSCNN has higher accuracy than state-of-the-art methods and has excellent noise immunity and transfer performance.

Artificial Intelligence for Surface-Enhanced Raman Spectroscopy.

Surface-enhanced Raman spectroscopy (SERS), well acknowledged as a fingerprinting and sensitive analytical technique, has exerted high applicational value in a broad range of fields including biomedicine, environmental protection, food safety among the others. In the endless pursuit of ever-sensitive, robust, and comprehensive sensing and imaging, advancements keep emerging in the whole pipeline of SERS, from the design of SERS substrates and reporter molecules, synthetic route planning, instrument refinement, to data preprocessing and analysis methods. Artificial intelligence (AI), which is created to imitate and eventually exceed human behaviors, has exhibited its power in learning high-level representations and recognizing complicated patterns with exceptional automaticity. Therefore, facing up with the intertwining influential factors and explosive data size, AI has been increasingly leveraged in all the above-mentioned aspects in SERS, presenting elite efficiency in accelerating systematic optimization and deepening understanding about the fundamental physics and spectral data, which far transcends human labors and conventional computations. In this review, the recent progresses in SERS are summarized through the integration of AI, and new insights of the challenges and perspectives are provided in aim to better gear SERS toward the fast track.

LayerUMAP: A tool for visualizing and understanding deep learning models in biological sequence classification using UMAP.

Despite the impressive success of deep learning techniques in various types of classification and prediction tasks, interpreting these models and explaining their predictions are still major challenges. In this article, we present an easy-to-use command line tool capable of visualizing and analyzing alternative representations of biological observations learned by deep learning models. This new tool, namely, layerUMAP, integrates autoBioSeqpy software and the UMAP library to address learned high-level representations. An important advantage of the tool is that it provides an interactive option that enables users to visualize the outputs of hidden layers along the depth of the model. We use two different classes of examples to illustrate the potential power of layerUMAP, and the results demonstrate that layerUMAP can provide insightful visual feedback about models and further guide us to develop better models.

Visual-semantic graph neural network with pose-position attentive learning for group activity recognition

Video-based group activities typically contain interactive contexts among diverse visual modalities between multiple persons, and semantic relationships between individual actions. Nevertheless, majority of the existing methods for recognizing group activity either captures the relationships among different persons by utilizing a solely RGB modality or neglect to exploit the label hierarchies between individual actions and the group activity. To tackle these issues, we propose a visual-semantic graph neural network, with pose-position attentive learning (VSGNN-PAL), for group activity recognition. Specifically, we first extract the individual-level appearance and motion representations from RGB and optical-flow inputs, to build a bi-modal visual graph. Two attentive aggregators are further proposed to integrate both the pose and position information to measure the relevance scores between persons, and dynamically refine the representation of each visual node from both modality-specific and cross-modal perspectives. To model a semantic hierarchy from a label space, we construct a semantic graph based on the linguistic embeddings of individual actions and group activity labels. We further employ a bi-directional mapping learning scheme, to integrate the label-relation-aware semantic context into the visual representations. Besides, a global reasoning module is introduced to progressively generate the group-level representations with the scene description maintained. Furthermore, we formulate a semantic-preserving loss, to maintain the consistency between the learned high-level representations and the semantics of the ground-truth labels. Experimental results on three group activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance.

Multi-view learning with distinguishable feature fusion for rumor detection

Researchers, enterprises, and governments have made great efforts to detect misinformation promptly and accurately. Traditional solutions either examine complicated hand-crafted features or rely heavily on the constructed credibility networks to extract useful indicators for discerning false information. However, such approaches require insightful domain expert knowledge and intensive feature engineering that are often non-generalizable. Recent advances in deep learning techniques have spurred learning high-level representations from textual and image content and discovering diffusion patterns with various neural networks. Despite the progress made by these methods, they still face the problem of overdependence on the content features and fail to discriminate against the influence of each user involved in the process of rumor spreading. Different user-aspect information plays different roles in various stages of rumor diffusion, effectively extract features from each aspect, and aggregate the learned features into a unique representation, which has not been well investigated. To address these limitations, we propose a novel model, UMLARD (User-aspect Multi-view Learning with Attention for Rumor Detection), to effectively learn the representation of different views of the users who engaged in spreading the tweet, and fuse the learned features through the distinguishable fusion mechanism. Finally, we concatenate the learned user-aspect features with content features to form a unique representation and feed it into a fully connected layer to predict the label of rumors. Our experiments conducted on real-world datasets demonstrate that UMLARD significantly improves the rumor detection performance compared to state-of-the-art baselines. It also allows explainability of the model behavior and the predicted results.

PFWNet: Pretraining neural network via feature jigsaw puzzle for weakly-supervised temporal action localization

Weakly supervised temporal action localization is a challenging yet interesting task. Existing methods usually apply a few temporal convolutional layers or linear layers to predict classification scores, where the model capacity is limited. Inspired by counterpart researches, increasing model capacity is the potential to improve the localization performance. However, under the weakly supervised paradigm, the video-level classification label is insufficient to learn large-capacity models. The essential reason lies in that most of the inputs to action localization networks are high-level features extracted by video recognition models. In lack of off-the-shelf initialization weights, the action localization networks have to train from scratch and can only explore low-capacity models. In this work, we are inspired by the self-supervised learning paradigm and propose to learn high-quality representative models via solving the feature jigsaw puzzle task. The proposed self-supervised pretraining process can explore networks with large kernel size and deeper layers, which can provide valuable initialization to action localization networks. In the implementation, we first discover potential action scopes via calculating motion intensity. Then, we cut features into snippets and permute them into out-of-order status. We randomly discard frames for boundaries between two snippets to guide the network learning high-level representations and prevent information leakage. Moreover, because the potential permutation number factorially rises with the increase of snippet number, we select a fixed number of permutation operations via the maximum hamming distance criterion, which eases the learning process. Comprehensive experiments on two benchmarks demonstrate the efficiency of pretraining to weakly supervised action localization task, and the proposed method builds new state-of-the-art performance.

Robust Feature Extraction for Geochemical Anomaly Recognition Using a Stacked Convolutional Denoising Autoencoder

Deep neural networks perform very well in learning high-level representations in support of multivariate geochemical anomaly recognition. Geochemical exploration data typically contain a proportion of large variations and missing values, which motivated us to construct a network architecture optimized to deal with these data. Our approach adopted a stacked convolutional denoising autoencoder (SCDAE) to extract robust features and decreased the level of sensitivity to partially corrupted data, that is, input data that are partially missing. SCDAE parameters, which include the network depth, number of convolution layers, number of convolution kernels, and convolution kernel size, were optimized using trial-and-error experiments. The optimal SCDAE architecture was then used to recognize multivariate geochemical anomalies related to mineralization in a case study in southwestern Fujian Province, based on the differences in the reconstruction errors between sample populations. The spatial distribution of high reconstruction errors in the anomaly map was closely related to most known Fe deposits, indicating the effectiveness of the SCDAE at recognizing geochemical anomalies related to Fe mineralization. A comparative study between the SCDAE and a stacked convolutional autoencoder (SCAE) with different corruption levels showed that the SCDAE exhibited reduced sensitivity to stochastic disturbances with different corruption proportions, and had an enhanced ability to recognize geochemical anomalies varying in a reasonable range. The robustness of the SCDAE makes it applicable to a wide variety of geochemical exploration scenarios, particularly in areas with incomplete or missing data.

Multiscale Convolutional Attention Network for Predicting Remaining Useful Life of Machinery

To integrate the complete degradation information of machinery, deep learning-based prognostics approaches usually use monitoring data acquired by different sensors as the inputs of networks. These approaches, however, lack an explicit learning mechanism to effectively identify the distinctions of different sensor data and highlight the important degradation information, thereby affecting the accuracy of deep prognostics networks and limiting their generalization. To overcome the aforementioned weaknesses, a new deep prognostics framework named multiscale convolutional attention network (MSCAN) is proposed in this article for predicting the remaining useful life (RUL) of machinery. In the proposed MSCAN, self-attention modules are first constructed to effectively fuse the input multisensor data. Then, a multiscale learning strategy is developed to automatically learn representations from different temporal scales. Finally, the learned high-level representations are fed into dynamic dense layers to perform regression analysis and RUL estimation. The proposed MSCAN is evaluated using multisensor monitoring data from life testing of milling cutters, and also compared with some state-of-the-art prognostics approaches. Experimental results demonstrate the effectiveness and superiority of the proposed MSCAN in fusing multisensor information and improving RUL prediction accuracy.

NHL Pathological Image Classification Based on Hierarchical Local Information and GoogLeNet-Based Representations.

Background Accurate classification for different non-Hodgkin lymphomas (NHL) is one of the main challenges in clinical pathological diagnosis due to its intrinsic complexity. Therefore, this paper proposes an effective classification model for three types of NHL pathological images, including mantle cell lymphoma (MCL), follicular lymphoma (FL), and chronic lymphocytic leukemia (CLL). Methods There are three main parts with respect to our model. First, NHL pathological images stained by hematoxylin and eosin (H&E) are transferred into blue ratio (BR) and Lab spaces, respectively. Then specific patch-level textural and statistical features are extracted from BR images and color features are obtained from Lab images both using a hierarchical way, yielding a set of hand-crafted representations corresponding to different image spaces. A random forest classifier is subsequently trained for patch-level classification. Second, H&E images are cropped and fed into a pretrained google inception net (GoogLeNet) for learning high-level representations and a softmax classifier is used for patch-level classification. Finally, three image-level classification strategies based on patch-level results are discussed including a novel method for calculating the weighted sum of patch results. Different classification results are fused at both feature 1 and image levels to obtain a more satisfactory result. Results The proposed model is evaluated on a public IICBU Malignant Lymphoma Dataset and achieves an improved overall accuracy of 0.991 and area under the receiver operating characteristic curve of 0.998. Conclusion The experimentations demonstrate the significantly increased classification performance of the proposed model, indicating that it is a suitable classification approach for NHL pathological images.

Identification of cancer subtypes by integrating multiple types of transcriptomics data with deep learning in breast cancer

Abstract The identification of cancer subtypes is vital to advance the precision of cancer disease diagnosis and therapy. Several works had been done to integrate multiple types of genomics data to investigate cancer subtypes. However, (1) few of them particularly considered the intrinsic correlations in each type of data; (2) to the best of our knowledge, none of them considered transcriptome alternative splicing regulation in data integration. It has been demonstrated that many cancers are related to abnormal alternative splicing regulations in recent years. In this paper, we propose a hierarchical deep learning framework, HI-SAE, to integrate gene expression and transcriptome alternative splicing profiles data to identify cancer subtypes. We adopt the stacked autoencoder (SAE) neural network to learn high-level representations in each type of data, respectively, and then integrate all the learned high-level representations by another learning layer to learn more complex data representations. Based on the final learned data representations, we cluster patients into different cancer subtype groups. Comprehensive experiments based on TCGA breast cancer data demonstrate that our model provides an effective and useful approach to integrate multiple types of transcriptomics data to identify cancer subtypes and the transcriptome alternative splicing data offers distinguishable clues of cancer subtypes.