Hierarchical Representation Research Articles

Hierarchical compositional representations for few-shot action recognition

Recently action recognition has received more and more attention for its comprehensive and practical applications in intelligent surveillance and human–computer interaction. However, few-shot action recognition has not been well explored and remains challenging because of data scarcity. In this paper, we propose a novel hierarchical compositional representations (HCR) learning approach for few-shot action recognition. Specifically, we divide a complicated action into several sub-actions by carefully designed hierarchical clustering and further decompose the sub-actions into more fine-grained spatially attentional sub-actions (SAS-actions). Although there exist large differences between base classes and novel classes, they can share similar patterns in sub-actions or SAS-actions. Furthermore, we adopt the Earth Mover’s Distance in the transportation problem to measure the similarity between video samples in terms of sub-action representations. It computes the optimal matching flows between sub-actions as distance metric, which is favorable for comparing fine-grained patterns. Extensive experiments show our method achieves the state-of-the-art results on HMDB51, UCF101 and Kinetics datasets.

Knowledge Graph-Based Hierarchical Text Semantic Representation

Document representation is the basis of language modeling. Its goal is to turn natural language text that flows into a structured form that can be stored and processed by a computer. The bag-of-words model is used by most of the text-representation methods that are currently available. And yet, they do not consider how phrases are used in the text, which hurts the performance of tasks that use natural language processing later on. Representing the meaning of text by phrases is a promising area of future research, but it is hard to do well because phrases are organized in a hierarchy and mining efficiency is low. In this paper, we put forward a method called hierarchical text semantic representation using the knowledge graph (HTSRKG), which uses syntactic structure features to find hierarchical phrases and knowledge graphs to improve how phrases are evaluated. First, we use CKY and PCFG to build the syntax tree sentence by sentence. Second, we walk through the parse tree using the hierarchical routing process to obtain the mixed phrase semantics in passages. Finally, the introduction of the knowledge graph improves the efficiency of text semantic extraction and the accuracy of text representation. This gives us a solid foundation for tasks involving natural language processing that come after. Extensive testing on actual datasets shows that HTSRKG surpasses baseline approaches with respect to text semantic representation, and the results of a recent benchmarking study support this.

Cross-language semantic-affective interaction – with evidence from Chinese EFL learners

Abstract Semantic and affective priming have long been treated separately in psycholinguistic studies. Recently, however, the question of whether and how these two primings interact has become controversial, especially in cross-language contexts where such discussions are rare. In the present study, four mixed-design experiments were conducted with Chinese EFL learners to investigate cross-language semantic-affective interactions: 3 (prime valence: negative, positive, neutral) × 2 (semantic relatedness: related, unrelated). Results show that semantic priming effects occurred in the L1L1 and L1L2 conditions, whereas affective priming effects were observed in the L2L2 condition. In the L2L1 priming condition, only emotion primes induced cross-language priming. These results suggest that semantic and emotional accesses are activated automatically and separately, but can facilitate cross-language word processing mutually. The results support the hierarchical representation of semantic features of emotion words from L1 to L2 in the unbalanced bilingual mental lexicon, while affective attributes are spread across a distributed network.

Dual-space Hierarchical Learning for Goal-guided Conversational Recommendation

Proactively and naturally guiding the dialog from the non-recommendation context (e.g., Chit-chat) to the recommendation scenario (e.g., Music) is crucial for the Conversational Recommender System (CRS). Prior studies mainly focus on planning the next dialog goal (e.g., chat on a movie star) conditioned on the previous dialog. However, we find the dialog goals can be simultaneously observed at different levels, which can be utilized to improve CRS. In this paper, we propose Dual-space Hierarchical Learning (DHL) to leverage multi-level goal sequences and their hierarchical relationships for conversational recommendation. Specifically, we exploit multi-level goal sequences from both the representation space and the optimization space. In the representation space, we propose the hierarchical representation learning where a cross attention module derives mutually enhanced multi-level goal representations. In the optimization space, we devise the hierarchical weight learning to reweight lower-level goal sequences, and introduce bi-level optimization for stable update. Additionally, we propose a soft labeling strategy to guide optimization gradually. Experiments on two real-world datasets verify the effectiveness of our approach. Code and data are available here.

Haisor : Human-aware Indoor Scene Optimization via Deep Reinforcement Learning

3D scene synthesis facilitates and benefits many real-world applications. Most scene generators focus on making indoor scenes plausible via learning from training data and leveraging extra constraints such as adjacency and symmetry. Although the generated 3D scenes are mostly plausible with visually realistic layouts, they can be functionally unsuitable for human users to navigate and interact with furniture. Our key observation is that human activity plays a critical role and sufficient free space is essential for human-scene interactions. This is exactly where many existing synthesized scenes fail—the seemingly correct layouts are often not fit for living. To tackle this, we present a human-aware optimization framework Haisor for 3D indoor scene arrangement via reinforcement learning, which aims to find an action sequence to optimize the indoor scene layout automatically. Based on the hierarchical scene graph representation, an optimal action sequence is predicted and performed via Deep Q-Learning with Monte Carlo Tree Search (MCTS), where MCTS is our key feature to search for the optimal solution in long-term sequences and large action space. Multiple human-aware rewards are designed as our core criteria of human-scene interaction, aiming to identify the next smart action by leveraging powerful reinforcement learning. Our framework is optimized end-to-end by giving the indoor scenes with part-level furniture layout including part mobility information. Furthermore, our methodology is extensible and allows utilizing different reward designs to achieve personalized indoor scene synthesis. Extensive experiments demonstrate that our approach optimizes the layout of 3D indoor scenes in a human-aware manner, which is more realistic and plausible than original state-of-the-art generator results, and our approach produces superior smart actions, outperforming alternative baselines.

Sparse and Hierarchical Transformer for Survival Analysis on Whole Slide Images.

The Transformer-based methods provide a good opportunity for modeling the global context of gigapixel whole slide image (WSI), however, there are still two main problems in applying Transformer to WSI-based survival analysis task. First, the training data for survival analysis is limited, which makes the model prone to overfitting. This problem is even worse for Transformer-based models which require large-scale data to train. Second, WSI is of extremely high resolution (up to 150,000 x 150,000 pixels) and is typically organized as a multi-resolution pyramid. Vanilla Transformer cannot model the hierarchical structure of WSI (such as patch cluster-level relationships), which makes it incapable of learning hierarchical WSI representation. To address these problems, in this paper, we propose a novel Sparse and Hierarchical Transformer (SH-Transformer) for survival analysis. Specifically, we introduce sparse self-attention to alleviate the overfitting problem, and propose a hierarchical Transformer structure to learn the hierarchical WSI representation. Experimental results based on three WSI datasets show that the proposed framework outperforms the state-of-the-art methods.

H3-Mapping: Quasi-Heterogeneous Feature Grids for Real-Time Dense Mapping Using Hierarchical Hybrid Representation

H3-Mapping: Quasi-Heterogeneous Feature Grids for Real-Time Dense Mapping Using Hierarchical Hybrid Representation

Hierarchical Sampling Representation Detector for Ship Detection in SAR Images

Hierarchical Sampling Representation Detector for Ship Detection in SAR Images

Predictive Clustering of Vessel Behavior Based on Hierarchical Trajectory Representation

Predictive Clustering of Vessel Behavior Based on Hierarchical Trajectory Representation

Conditional Goal-Oriented Trajectory Prediction for Interacting Vehicles.

Predicting future trajectories of pairwise traffic agents in highly interactive scenarios, such as cut-in, yielding, and merging, is challenging for autonomous driving. The existing works either treat such a problem as a marginal prediction task or perform single-axis factorized joint prediction, where the former strategy produces individual predictions without considering future interaction, while the latter strategy conducts conditional trajectory-oriented prediction via agentwise interaction or achieves conditional rollout-oriented prediction via timewise interaction. In this article, we propose a novel double-axis factorized joint prediction pipeline, namely, conditional goal-oriented trajectory prediction (CGTP) framework, which models future interaction both along the agent and time axes to achieve goal and trajectory interactive prediction. First, a goals-of-interest network (GoINet) is designed to extract fine-grained features of goal candidates via hierarchical vectorized representation. Furthermore, we propose a conditional goal prediction network (CGPNet) to produce multimodal goal pairs in an agentwise conditional manner, along with a newly designed goal interactive loss to better learn the joint distribution of the intermediate interpretable modes. Explicitly guided by the goal-pair predictions, we propose a goal-oriented trajectory rollout network (GTRNet) to predict scene-compliant trajectory pairs via timewise interactive rollouts. Extensive experimental results confirm that the proposed CGTP outperforms the state-of-the-art (SOTA) prediction models on the Waymo open motion dataset (WOMD), Argoverse motion forecasting dataset, and In-house cut-in dataset. Code is available at https://github.com/LiDinga/CGTP/.

HKTSG: A Hierarchical Knowledge-guided Traffic Scene Graph Representation Learning Framework for Intelligent Vehicles

HKTSG: A Hierarchical Knowledge-guided Traffic Scene Graph Representation Learning Framework for Intelligent Vehicles

Semantics-Guided Hierarchical Feature Encoding Generative Adversarial Network for Visual Image Reconstruction From Brain Activity.

The utilization of deep learning techniques for decoding visual perception images from brain activity recorded by functional magnetic resonance imaging (fMRI) has garnered considerable attention in recent research. However, reconstructed images from previous studies still suffer from low quality or unreliability. Moreover, the complexity inherent to fMRI data, characterized by high dimensionality and low signal-to-noise ratio, poses significant challenges in extracting meaningful visual information for perceptual reconstruction. In this regard, we proposes a novel neural decoding model, named the hierarchical semantic generative adversarial network (HS-GAN), inspired by the hierarchical encoding of the visual cortex and the homology theory of convolutional neural networks (CNNs), which is capable of reconstructing perceptual images from fMRI data by leveraging the hierarchical and semantic representations. The experimental results demonstrate that HS-GAN achieved the best performance on Horikawa2017 dataset (histogram similarity: 0.447, SSIM-Acc: 78.9%, Peceptual-Acc: 95.38%, AlexNet(2): 96.24% and AlexNet(5): 94.82%) over existing advanced methods, indicating improved naturalness and fidelity of the reconstructed image. The versatility of the HS-GAN was also highlighted, as it demonstrated promising generalization capabilities in reconstructing handwritten digits, achieving the highest SSIM (0.783±0.038), thus extending its application beyond training solely on natural images.

Hierarchical path planning from speech instructions with spatial concept-based topometric semantic mapping.

Assisting individuals in their daily activities through autonomous mobile robots is a significant concern, especially for users without specialized knowledge. Specifically, the capability of a robot to navigate to destinations based on human speech instructions is crucial. Although robots can take different paths toward the same objective, the shortest path is not always the most suitable. A preferred approach would be to accommodate waypoint specifications flexibly for planning an improved alternative path even with detours. Furthermore, robots require real-time inference capabilities. In this sense, spatial representations include semantic, topological, and metric-level representations, each capturing different aspects of the environment. This study aimed to realize a hierarchical spatial representation using a topometric semantic map and path planning with speech instructions by including waypoints. Thus, we present a hierarchical path planning method called spatial concept-based topometric semantic mapping for hierarchical path planning (SpCoTMHP), which integrates place connectivity. This approach provides a novel integrated probabilistic generative model and fast approximate inferences with interactions among the hierarchy levels. A formulation based on "control as probabilistic inference" theoretically supports the proposed path planning algorithm. We conducted experiments in a home environment using the Toyota human support robot on the SIGVerse simulator and in a lab-office environment with the real robot Albert. Here, the user issues speech commands that specify the waypoint and goal, such as "Go to the bedroom via the corridor." Navigation experiments were performed using speech instructions with a waypoint to demonstrate the performance improvement of the SpCoTMHP over the baseline hierarchical path planning method with heuristic path costs (HPP-I) in terms of the weighted success rate at which the robot reaches the closest target (0.590) and passes the correct waypoints. The computation time was significantly improved by 7.14 s with the SpCoTMHP than the baseline HPP-I in advanced tasks. Thus, hierarchical spatial representations provide mutually understandable instruction forms for both humans and robots, thus enabling language-based navigation.

Deep Hierarchical Multimodal Metric Learning.

Multimodal metric learning aims to transform heterogeneous data into a common subspace where cross-modal similarity computing can be directly performed and has received much attention in recent years. Typically, the existing methods are designed for nonhierarchical labeled data. Such methods fail to exploit the intercategory correlations in the label hierarchy and, therefore, cannot achieve optimal performance on hierarchical labeled data. To address this problem, we propose a novel metric learning method for hierarchical labeled multimodal data, named deep hierarchical multimodal metric learning (DHMML). It learns the multilayer representations for each modality by establishing a layer-specific network corresponding to each layer in the label hierarchy. In particular, a multilayer classification mechanism is introduced to enable the layerwise representations to not only preserve the semantic similarities within each layer, but also retain the intercategory correlations across different layers. In addition, an adversarial learning mechanism is proposed to bridge the cross-modality gap by producing indistinguishable features for different modalities. Through integration of the multilayer classification and adversarial learning mechanisms, DHMML can obtain hierarchical discriminative modality-invariant representations for multimodal data. Experiments on two benchmark datasets are used to demonstrate the superiority of the proposed DHMML method over several state-of-the-art methods.

GenHPF: General Healthcare Predictive Framework for Multi-task Multi-source Learning.

Despite the remarkable progress in the development of predictive models for healthcare, applying these algorithms on a large scale has been challenging. Algorithms trained on a particular task, based on specific data formats available in a set of medical records, tend to not generalize well to other tasks or databases in which the data fields may differ. To address this challenge, we propose General Healthcare Predictive Framework (GenHPF), which is applicable to any EHR with minimal preprocessing for multiple prediction tasks. GenHPF resolves heterogeneity in medical codes and schemas by converting EHRs into a hierarchical textual representation while incorporating as many features as possible. To evaluate the efficacy of GenHPF, we conduct multi-task learning experiments with single-source and multi-source settings, on three publicly available EHR datasets with different schemas for 12 clinically meaningful prediction tasks. Our framework significantly outperforms baseline models that utilize domain knowledge in multi-source learning, improving average AUROC by 1.2%P in pooled learning and 2.6%P in transfer learning while also showing comparable results when trained on a single EHR dataset. Furthermore, we demonstrate that self-supervised pretraining using multi-source datasets is effective when combined with GenHPF, resulting in a 0.6 pretraining. By eliminating the need for preprocessing and feature engineering, we believe that this work offers a solid framework for multi-task and multi-source learning that can be leveraged to speed up the scaling and usage of predictive algorithms in healthcare.1.

Training and testing an artificial intelligence model for action recognition using the MMAction2 toolkit

In recent decades, artificial intelligence (AI) has become an indispensable tool in various fields of human activity, including action recognition. With the advancements in machine learning and deep learning technologies, AI's capabilities in analyzing and interpreting actions performed by humans or objects in video and audio recordings have significantly increased. This progress has led to the development of numerous applications such as surveillance systems, human-computer interaction, sports analytics, and autonomous driving, where understanding and recognizing actions is crucial. Traditional methods of action recognition relied heavily on handcrafted features and classical machine learning algorithms, which often struggled with the variability and complexity of real-world scenarios. The emergence of deep learning, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs), has revolutionized this field by enabling models to learn hierarchical representations directly from raw data, thus improving performance and robustness. In this paper, we describe the process of training and testing an artificial intelligence model in an action recognition task using the MMAction2 toolkit.

Artificial Bee Colony Algorithm-based Feature Selection and Hybrid ML Framework for Efficient Rice Yield Prediction

India's economy predominantly depends on monsoon and agricultural output. Agribusiness products contribute to nearly a quarter of its gross domestic product and 58% of its population depends on agriculture for their livelihood. Certain crops, like rice, are vital to its food security being the most widely grown crop and accounting for one-third production of foodgrains in India. Understanding and enhancing its production is critical in ensuring food availability and promoting sustainable agricultural practices. Rice yield prediction has been a most researched area in the agriculture domain. Machine Learning (ML) frameworks have been found to perform well in patches with large, complex datasets as insufficient feature engineering and temporal dependencies plague efficacy. In this paper, we propose a swam-based meta-heuristic artificial bee colony (ABC) algorithm for feature selection from the dataset sourced from the Agricultural Production and Statistical Division of the Department of Agriculture Cooperation and Farmers Welfare, Government of India. The feature engineering is further optimized by a hybrid model comprising a convolutional neural network (CNN) for learning hierarchical representations and identifying relevant attributes from the complex dataset and long short-term memory (LSTM) for temporal aspects. Finally, a random forest (RF) regressor provides the benefits of ensemble learning, which merges multiple decision trees to remove bias, and variance and improve prediction accuracy. From the results, it is observed that the proposed hybrid model outperforms existing state-of-the-art standalone and hybrid models with the highest coefficient of determination (R2) and lowest mean square error (MSE) of 0.989 and 13613 respectively. The reliable and efficient hybrid model can aid farmers and policymakers in making informed decisions related to rice yield prediction leading to sustainable agricultural practices.

Scalable Face Image Coding via StyleGAN Prior: Toward Compression for Human-Machine Collaborative Vision.

The accelerated proliferation of visual content and the rapid development of machine vision technologies bring significant challenges in delivering visual data on a gigantic scale, which shall be effectively represented to satisfy both human and machine requirements. In this work, we investigate how hierarchical representations derived from the advanced generative prior facilitate constructing an efficient scalable coding paradigm for human-machine collaborative vision. Our key insight is that by exploiting the StyleGAN prior, we can learn three-layered representations encoding hierarchical semantics, which are elaborately designed into the basic, middle, and enhanced layers, supporting machine intelligence and human visual perception in a progressive fashion. With the aim of achieving efficient compression, we propose the layer-wise scalable entropy transformer to reduce the redundancy between layers. Based on the multi-task scalable rate-distortion objective, the proposed scheme is jointly optimized to achieve optimal machine analysis performance, human perception experience, and compression ratio. We validate the proposed paradigm's feasibility in face image compression. Extensive qualitative and quantitative experimental results demonstrate the superiority of the proposed paradigm over the latest compression standard Versatile Video Coding (VVC) in terms of both machine analysis as well as human perception at extremely low bitrates (< 0.01 bpp), offering new insights for human-machine collaborative compression.

Sentiment Analysis on Twitter Using Deep Belief Network Optimized with Particle Swarm Optimization

Deep Belief Network is a type of artificial neural network that is widely used in machine learning and deep learning tasks that allows it to learn hierarchical representations of the input data. However, Deep Belief Network has a drawback of being sensitive to hyperparameters. DBN has several hyperparameters that need to be chosen appropriately for the network to function effectively. Poor hyperparameter choices can lead to unstable training or poor performance. Therefore, the Particle Swarm Optimization algorithm is used to search for the best hyperparameters, which can lead to stable training and improved performance. The purpose of this study is to analyze public sentiment on Twitter using the Deep Belief Network method and to optimize it using Particle Swarm Optimization. The evaluation results obtained in this study are 71.4% accuracy, 71.7% precision, 71.4% recall, and 71.2% F1-score in the Deep Belief Network model which is optimized by the Particle Swarm Optimization algorithm, whereas when compared with the Deep Belief Network model alone gets evaluation results of 68.3% Accuracy, 69.0% Precision, 68.3% Recall and 68.0% F1-score. These results indicate that the use of the Particle Swarm Optimization algorithm is quite influential in analyzing sentiment in text.

Exploiting the Role of Features for Antigens-Antibodies Interaction Site Prediction.

Antibodies are a class of proteins that recognize and neutralize pathogens by binding to their antigens. They are the most significant category of biopharmaceuticals for both diagnostic and therapeutic applications. Understanding how antibodies interact with their antigens plays a fundamental role in drug and vaccine design and helps to comprise the complex antigen binding mechanisms. Computational methods for predicting interaction sites of antibody-antigen are of great value due to the overall cost of experimental methods. Machine learning methods and deep learning techniques obtained promising results.In this work, we predict antibody interaction interface sites by applying HSS-PPI, a hybrid method defined to predict the interface sites of general proteins. The approach abstracts the proteins in terms of hierarchical representation and uses a graph convolutional network to classify the amino acids between interface and non-interface. Moreover, we also equipped the amino acids with different sets of physicochemical features together with structural ones to describe the residues. Analyzing the results, we observe that the structural features play a fundamental role in the amino acid descriptions. We compare the obtained performances, evaluated using standard metrics, with the ones obtained with SVM with 3D Zernike descriptors, Parapred, Paratome, and Antibody i-Patch.