Hierarchical Representation Learning Research Articles

Decontextualized learning for interpretable hierarchical representations of visual patterns

SummaryApart from discriminative modeling, the application of deep convolutional neural networks to basic research utilizing natural imaging data faces unique hurdles. Here, we present decontextualized hierarchical representation learning (DHRL), designed specifically to overcome these limitations. DHRL enables the broader use of small datasets, which are typical in most studies. It also captures spatial relationships between features, provides novel tools for investigating latent variables, and achieves state-of-the-art disentanglement scores on small datasets. DHRL is enabled by a novel preprocessing technique inspired by generative model chaining and an improved ladder network architecture and regularization scheme. More than an analytical tool, DHRL enables novel capabilities for virtual experiments performed directly on a latent representation, which may transform the way we perform investigations of natural image features, directly integrating analytical, empirical, and theoretical approaches.

Learning hierarchical face representation to enhance HCI among medical robots

In this paper, we propose a hierarchical framework for face recognition by learning deep representation. In order to exploit key patches for face recognition, we separate the entire image into several patches including eyes, nose, and mouth. A binary facegrid is generated to indicate the accurate position of the key patches in face image. The patches are fed into the hierarchical framework to learn the deep representation of the image. We leverage the PCA and SVM method for face recognition. Our face representation can enhance many medical robot applications. Comprehensive experiments have demonstrated that our proposed method can effectively recognize real human faces from fake samples.

Learning Hierarchical Review Graph Representations for Recommendation

The user review data have been demonstrated to be effective in solving different recommendation problems. Previous review-based recommendation methods usually employ sophisticated compositional models, such as Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN), to learn semantic representations from the review data for recommendation. However, these methods mainly capture the local dependency between neighbouring words in a word window, and they treat each review equally. Therefore, they may not be effective in capturing the global dependency between words, and tend to be easily biased by noise review information. In this paper, we propose a novel review-based recommendation model, named Review Graph Neural Network (RGNN). Specifically, RGNN builds a specific review graph for each individual user/item, which provides a global view about the user/item properties to help weaken the biases caused by noise review information. A type-aware graph attention mechanism is developed to learn semantic embeddings of words. Moreover, a personalized graph pooling operator is proposed to learn hierarchical representations of the review graph to form the semantic representation for each user/item. We compared RGNN with state-of-the-art review-based recommendation approaches on two real-world datasets. The experimental results indicate that RGNN consistently outperforms baseline methods, in terms of Mean Square Error (MSE).

Hierarchical Multi-View Graph Pooling with Structure Learning

Graph Neural Networks (GNNs), which generalize deep neural networks to graph-structured data, have drawn considerable attention and achieved state-of-the-art performance in numerous graph related tasks. However, existing GNN models mainly focus on designing graph convolution operations. The graph pooling (or downsampling) operations, that play an important role in learning hierarchical representations, are usually overlooked. In this paper, we proposed a novel multi-view graph pooling operator dubbed as MVPool, which ranks nodes across different views with different contextual graph information. Meanwhile, attention mechanism is utilized to promote the collaboration of different views for generating robust node rankings. Then the pooling operation adaptively selects a subset of nodes to form an induced subgraph based on the ranking list. To preserve the underlying graph topological information, we further introduce a structure learning mechanism to learn a refined graph structure for the pooled graph at each layer. The proposed MVPool operator is a general strategy that can be integrated into various graph neural network architectures. By combining MVPool operator with graph neural networks, we perform hierarchical representation learning for both node and graph level classification as well as clustering tasks. Experimental results on nine widely used benchmarks demonstrate the effectiveness of our proposed model.

Hierarchical Representation Learning for Attributed Networks

Network representation learning, also called network embedding, aiming to learn low dimensional vectors for nodes while preserving essential properties of the network, benefits plenty of practical applications. However, how to do representation learning on the network quickly and effectively is a meaningful and challenging task, especially for the attributed networks. In this paper, we propose HANE, a Hierarchical Attributed Network Embedding framework, which is a fast and effective method by quickly constructing a hierarchical attributed network of different granularities to learn nodes representations. Specifically, for an attributed network, HANE first builds a hierarchy of successively smaller attributed network from fine to coarse by the fast granulation strategy fusing topological structure and node attributes. After using any unsupervised network embedding method to learn nodes representations of the coarsest network, HANE refines the nodes representations of the hierarchical attributed network from coarse to fine. HANE improves the speed of network representation learning while maintaining its performance and the representation learning method of the coarsest network is flexible. We conduct extensive evaluations for the proposed framework HANE on six datasets and two benchmark applications. Experimental results demonstrate that HANE achieves significant improvements over previous state-of-the-art network embedding methods in efficiency and effectiveness.

Multiscale Representation Learning of Graph Data With Node Affinity

Graph neural networks have emerged as a popular and powerful tool for learning hierarchical representation of graph data. In complement to graph convolution operators, graph pooling is crucial for extracting hierarchical representation of data in graph neural networks. However, most recent graph pooling methods still fail to efficiently exploit the geometry of graph data. In this paper, we propose a novel graph pooling strategy that leverages node affinity to improve the hierarchical representation learning of graph data. Node affinity is computed by harmonizing the kernel representation of topology information and node features. In particular, a structure-aware kernel representation is introduced to explicitly exploit advanced topological information for efficient graph pooling without eigendecomposition of the graph Laplacian. Similarities of node signals are evaluated using the Gaussian radial basis function (RBF) in an adaptive way. Experimental results demonstrate that the proposed graph pooling strategy is able to achieve state-of-the-art performance on a collection of public graph classification benchmark datasets.

Sequence-to-Sequence Video Prediction by Learning Hierarchical Representations

Video prediction which maps a sequence of past video frames into realistic future video frames is a challenging task because it is difficult to generate realistic frames and model the coherent relationship between consecutive video frames. In this paper, we propose a hierarchical sequence-to-sequence prediction approach to address this challenge. We present an end-to-end trainable architecture in which the frame generator automatically encodes input frames into different levels of latent Convolutional Neural Network (CNN) features, and then recursively generates future frames conditioned on the estimated hierarchical CNN features and previous prediction. Our design is intended to automatically learn hierarchical representations of video and their temporal dynamics. Convolutional Long Short-Term Memory (ConvLSTM) is used in combination with skip connections so as to separately capture the sequential structures of multiple levels of hierarchy of features. We adopt Scheduled Sampling for training our recurrent network in order to facilitate convergence and to produce high-quality sequence predictions. We evaluate our method on the Bouncing Balls, Moving MNIST, and KTH human action dataset, and report favorable results as compared to existing methods.

Hierarchical extreme learning machine with L21-norm loss and regularization

Recently, multilayer extreme learning machine (ELM) algorithms have been extensively studied for hierarchical abstract representation learning in the ELM community. In this paper, we investigate the specific combination of $$L_{21}$$ -norm based loss function and regularization to improve the robustness and the sparsity of multilayer ELM. As we all known, the mean square error (MSE) cost function (or squared $$L_{2}$$ -norm cost function) is commonly used as optimization cost function for ELM, but it is sensitive to outliers and impulsive noises that are pervasive in real-world data. Our $$L_{21}$$ -norm loss function can lessen the harmful influence caused by noises and outliers and enhance robustness and stability of the learned model. Additionally, the row sparse inducing $$L_{21}$$ -norm regularization can learn the most-relevant sparse representation and reduce the intrinsic complexity of the learning model. We propose a specific combination of $$L_{21}$$ -norm loss function and regularization ELM auto-encoder (LR21-ELM-AE), and then stack LR21-ELM-AE hierarchically to construct the hierarchical extreme learning machine (H-LR21-ELM). Experiments conducted on several well-known benchmark datasets are presented, the results show that the proposed H-LR21-ELM can generate a more robust, more discriminative and sparser model compared with the other state-of-the-art multilayer ELM algorithms.

Fine-Grained Privacy Detection with Graph-Regularized Hierarchical Attentive Representation Learning

Due to the complex and dynamic environment of social media, user generated contents (UGCs) may inadvertently leak users’ personal aspects, such as the personal attributes, relationships and even the health condition, and thus place users at high privacy risks. Limited research efforts, thus far, have been dedicated to the privacy detection from users’ unstructured data (i.e., UGCs). Moreover, existing efforts mainly focus on applying conventional machine learning techniques directly to traditional hand-crafted privacy-oriented features, ignoring the powerful representing capability of the advanced neural networks. In light of this, in this article, we present a fine-grained privacy detection network (GrHA) equipped with graph-regularized hierarchical attentive representation learning. In particular, the proposed GrHA explores the semantic correlations among personal aspects with graph convolutional networks to enhance the regularization for the UGC representation learning, and, hence, fulfil effective fine-grained privacy detection. Extensive experiments on a real-world dataset demonstrate the superiority of the proposed model over state-of-the-art competitors in terms of eight standard metrics. As a byproduct, we have released the codes and involved parameters to facilitate the research community.

Hierarchical and Unsupervised Graph Representation Learning with Loukas’s Coarsening

We propose a novel algorithm for unsupervised graph representation learning with attributed graphs. It combines three advantages addressing some current limitations of the literature: (i) The model is inductive: it can embed new graphs without re-training in the presence of new data; (ii) The method takes into account both micro-structures and macro-structures by looking at the attributed graphs at different scales; (iii) The model is end-to-end differentiable: it is a building block that can be plugged into deep learning pipelines and allows for back-propagation. We show that combining a coarsening method having strong theoretical guarantees with mutual information maximization suffices to produce high quality embeddings. We evaluate them on classification tasks with common benchmarks of the literature. We show that our algorithm is competitive with state of the art among unsupervised graph representation learning methods.

Domain Adversarial Transfer Network for Cross-Domain Fault Diagnosis of Rotary Machinery

Recently, deep learning-based intelligent fault diagnosis techniques have obtained good classification performance with amount of supervised training data. However, domain shift problem between the training and testing data usually occurs due to variation in operating conditions and interferences of environment noise. Transfer learning provides a promising tool for handling the cross-domain diagnosis problems by leveraging knowledge from the source domain to help learning in the target domain. Most existing studies attempt to learn both domain features in a common feature space to reduce the domain shift, which are not optimal on specific discriminative tasks and can be limited to small shifts. This article proposes a novel domain adversarial transfer network (DATN), exploiting task-specific feature learning networks and domain adversarial training techniques for handling large distribution discrepancy across domains. First, two asymmetric encoder networks integrating deep convolutional neural networks are designed for learning hierarchical representations from the source domain and target domain. Then, the network weights learned in source tasks are transferred to improve training on target tasks. Finally, domain adversarial training with inverted label loss is introduced to minimize the difference between source and target distributions. To validate the effectiveness and superiority of the proposed method in the presence of large domain shifts, two fault data sets from different test rigs are investigated, and different fault severities, compound faults, and data contaminated by noise are considered. The experimental results demonstrate that the proposed method achieves the average accuracy of 96.45% on the bearing data set and 98.92% on the gearbox data set, which outperforms other algorithms.

Data‐driven nonlinear chemical process fault diagnosis based on hierarchical representation learning

AbstractRepresentation extraction is crucial in data‐driven process monitoring, and deep neural network (DNN) is an efficient tool for extracting representations from considerable process data. This study proposes a hierarchical representation learning (HRL) method that integrates the deep belief neural (DBN) network and support vector data description (SVDD) for efficient nonlinear chemical process fault diagnosis. First, hierarchical representations containing meaningful process information are generated through a DBN network by utilizing generally massive normal operating process data. Second, an SVDD‐based decision‐making system is constructed using generally small‐sized faulty data. Three experimental studies are then conducted. A comparison of results with those of several state‐of‐the‐art methods reveal the suitability of the HRL method for process monitoring due to its two main advantages. First, DNN has a superior representative ability and generates representations with richer process information than conventional data‐driven methods. Second, the HRL method utilizes available process data and is suitable for practical conditions in which considerable normal operating data but limited small‐sized faulty data are available.

Modeling Long-Term Dependencies from Videos Using Deep Multiplicative Neural Networks

Understanding temporal dependencies of videos is fundamental for vision problems, but deep learning–based models are still insufficient in this field. In this article, we propose a novel deep multiplicative neural network (DMNN) for learning hierarchical long-term representations from video. The DMNN is built upon the multiplicative block that remembers the pairwise transformations between consecutive frames using multiplicative interactions rather than the regular weighted-sum ones. The block is slided over the timesteps to update the memory of the networks on the frame pairs. Deep architecture can be implemented by stacking multiple layers of the sliding blocks. The multiplicative interactions lead to exact, rather than approximate, modeling of temporal dependencies. The memory mechanism can remember the temporal dependencies for an arbitrary length of time. The multiple layers output multiple-level representations that reflect the multi-timescale structure of video. Moreover, to address the difficulty of training DMNNs, we derive a theoretically sound convergent method, which leads to a fast and stable convergence. We demonstrate a new state-of-the-art classification performance with proposed networks on the UCF101 dataset and the effectiveness of capturing complicate temporal dependencies on a variety of synthetic datasets.

ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations

Graph Neural Networks (GNN) have been shown to work effectively for modeling graph structured data to solve tasks such as node classification, link prediction and graph classification. There has been some recent progress in defining the notion of pooling in graphs whereby the model tries to generate a graph level representation by downsampling and summarizing the information present in the nodes. Existing pooling methods either fail to effectively capture the graph substructure or do not easily scale to large graphs. In this work, we propose ASAP (Adaptive Structure Aware Pooling), a sparse and differentiable pooling method that addresses the limitations of previous graph pooling architectures. ASAP utilizes a novel self-attention network along with a modified GNN formulation to capture the importance of each node in a given graph. It also learns a sparse soft cluster assignment for nodes at each layer to effectively pool the subgraphs to form the pooled graph. Through extensive experiments on multiple datasets and theoretical analysis, we motivate our choice of the components used in ASAP. Our experimental results show that combining existing GNN architectures with ASAP leads to state-of-the-art results on multiple graph classification benchmarks. ASAP has an average improvement of 4%, compared to current sparse hierarchical state-of-the-art method. We make the source code of ASAP available to encourage reproducible research 1.

Learning Hierarchical Representations of Stories by Using Multi-layered Structures in Narrative Multimedia.

Narrative works (e.g., novels and movies) consist of various utterances (e.g., scenes and episodes) with multi-layered structures. However, the existing studies aimed to embed only stories in a narrative work. By covering other granularity levels, we can easily compare narrative utterances that are coarser (e.g., movie series) or finer (e.g., scenes) than a narrative work. We apply the multi-layered structures on learning hierarchical representations of the narrative utterances. To represent coarser utterances, we consider adjacency and appearance of finer utterances in the coarser ones. For the movies, we suppose a four-layered structure (character roles ∈ characters ∈ scenes ∈ movies) and propose three learning methods bridging the layers: Char2Vec, Scene2Vec, and Hierarchical Story2Vec. Char2Vec represents a character by using dynamic changes in the character’s roles. To find the character roles, we use substructures of character networks (i.e., dynamic social networks of characters). A scene describes an event. Interactions between characters in the scene are designed to describe the event. Scene2Vec learns representations of a scene from interactions between characters in the scene. A story is a series of events. Meanings of the story are affected by order of the events as well as their content. Hierarchical Story2Vec uses sequential order of scenes to represent stories. The proposed model has been evaluated by estimating the similarity between narrative utterances in real movies.

Improved Highway Network Block for Training Very Deep Neural Networks

Very deep networks are successful in various tasks with reported results surpassing human performance. However, training such very deep networks is not trivial. Typically, the problems of learning the identity function and feature reuse can work together to plague optimization of very deep networks. In this paper, we propose a highway network with gate constraints that addresses the aforementioned problems, and thus alleviates the difficulty of training. Namely, we propose two variants of highway network, HWGC and HWCC , employing feature summation and concatenation respectively. The proposed highway networks, besides being more computationally efficient, are shown to have more interesting learning characteristics such as natural learning of hierarchical and robust representations due to a more effective usage of model depth, fewer gates for successful learning, better generalization capacity and faster convergence than the original highway network. Experimental results show that our models outperform the original highway network and many state-of-the-art models. Importantly, we observe that our second model with feature concatenation and compression consistently outperforms our model with feature summation of similar depth, the original highway network, many state-of-the-art models and even ResNets on four benchmarking datasets which are CIFAR-10, CIFAR-100, Fashion-MNIST, SVHN and imagenet-2012 (ILSVRC) datasets. Furthermore, the second proposed model is more computationally efficient than the state-of-the-art in view of training, inference time and GPU memory resource, which strongly supports real-time applications. Using a similar number of model parameters for the CIFAR-10, CIFAR-100, Fashion-MNIST and SVHN datasets, the significantly shallower proposed model can surpass the performance of ResNet-110 and ResNet-164 that are roughly 6 and 8 times deeper, respectively. Similarly, for the imagenet dataset, the proposed models surpass the performance of ResNet-101 and ResNet-152 that are roughly three times deeper.

Deep nonsmooth nonnegative matrix factorization network with semi-supervised learning for SAR image change detection

In the paper, we propose a deep nonsmooth nonnegative matrix factorization (nsNMF) network with semi-supervised learning for synthetic aperture radar (SAR) image change detection. In most of the existing deep-NMF-based models, the nonnegative matrix is linearly decomposed layer by layer, which may fail to characterize the nonlinearities in complex data. As such, a nonlinear deep nsNMF model is first built for learning hierarchical, nonlinear, and localized data representations. Meanwhile, in view of its good generalization performance and low computational complexity, extreme learning machine (ELM) is integrated into the nonlinear deep nsNMF model to construct a deep nsNMF network for satisfactory classification. More importantly, since it is difficult to acquire more labeled samples in practice, semi-supervised learning strategy is proposed to make use of partially labeled data for training. The learning process of the proposed network consists of pretraining stage and fine-tuning stage, in which the former pretrains all decomposed matrices layer by layer and the latter aims to reduce the total reconstruction error by using the mini-batch gradient descent algorithm. The experimental results on four pairs of SAR images demonstrate the effectiveness of the proposed method.

Representation learning of genomic sequence motifs with convolutional neural networks.

Although convolutional neural networks (CNNs) have been applied to a variety of computational genomics problems, there remains a large gap in our understanding of how they build representations of regulatory genomic sequences. Here we perform systematic experiments on synthetic sequences to reveal how CNN architecture, specifically convolutional filter size and max-pooling, influences the extent that sequence motif representations are learned by first layer filters. We find that CNNs designed to foster hierarchical representation learning of sequence motifs—assembling partial features into whole features in deeper layers—tend to learn distributed representations, i.e. partial motifs. On the other hand, CNNs that are designed to limit the ability to hierarchically build sequence motif representations in deeper layers tend to learn more interpretable localist representations, i.e. whole motifs. We then validate that this representation learning principle established from synthetic sequences generalizes to in vivo sequences.

Can deep learning predict risky retail investors? A case study in financial risk behavior forecasting

The paper examines the potential of deep learning to support decisions in financial risk management. We develop a deep learning model for predicting whether individual spread traders secure profits from future trades. This task embodies typical modeling challenges faced in risk and behavior forecasting. Conventional machine learning requires data that is representative of the feature-target relationship and relies on the often costly development, maintenance, and revision of handcrafted features. Consequently, modeling highly variable, heterogeneous patterns such as trader behavior is challenging. Deep learning promises a remedy. Learning hierarchical distributed representations of the data in an automatic manner (e.g. risk taking behavior), it uncovers generative features that determine the target (e.g., trader’s profitability), avoids manual feature engineering, and is more robust toward change (e.g. dynamic market conditions). The results of employing a deep network for operational risk forecasting confirm the feature learning capability of deep learning, provide guidance on designing a suitable network architecture and demonstrate the superiority of deep learning over machine learning and rule-based benchmarks.

Deep Learning Hierarchical Representation From Heterogeneous Flow-Level Communication Data

The success of a detection model depends heavily on feature engineering. Deep learning has been successfully applied in numerous research fields as a universal representation learning method. However, the heterogeneity of flow-level communication data obstructs the application of deep learning to communication representation learning, and research on this problem is still lacking. To cope with this problem, we propose a heterogeneous communication data-encoding approach to extract fixed-size encoding data to apply deep learning to heterogeneous communication data by preserving the spatiotemporal characteristics of the data. Then, we propose a feature extractor based on deep learning to automatically learn hierarchical and robust communication representations without expert knowledge. We show that the proposed approach can replicate and optimize the key steps of feature engineering well and learn hierarchical representations directly from heterogeneous communication data. Moreover, compared with features extracted with principal component analysis (PCA), manifold learning and manually crafted methods, the features extracted by deep learning are more robust and are characterized by their better adaptability to various classifiers and datasets. To the best of our knowledge, the initial work here is the first to apply deep learning techniques to heterogeneous flow-level data; consequently, the heterogeneous communication data processing method can provide technical means for the application of deep learning in other communication-related research fields.