Hierarchical Semantic Information Research Articles

Pseudo-label refinement via hierarchical contrastive learning for source-free unsupervised domain adaptation

Source-free unsupervised domain adaptation aims to adapt a source model to an unlabeled target domain without accessing the source data due to privacy considerations. Existing works mainly solve the problem by self-training methods and representation learning. However, these works typically learn the representation on a single semantic level and barely exploit the rich hierarchical semantic information to obtain clear decision boundaries, which makes it hard for these methods to achieve satisfactory generalization performance. In this paper, we propose a novel hierarchical contrastive domain adaptation algorithm that exploits self-supervised contrastive learning on both fine-grained instances and coarse-grained cluster semantics. On the one hand, we propose an adaptive prototype pseudo-labeling strategy to obtain much more reliable labels. On the other hand, we propose hierarchical contrastive representation learning on both fine-grained instance-wise level and coarse-grained cluster level to reduce the negative effect of label noise and stabilize the whole training procedure. Extensive experiments are conducted on primary unsupervised domain adaptation benchmark datasets, and the results demonstrate the effectiveness of the proposed method.

Multi-task hierarchical convolutional network for visual-semantic cross-modal retrieval

Bridging visual and textual representations plays a central role in delving into multimedia data understanding. The main challenge arises from that images and texts exist in heterogeneous spaces, leading to the difficulty to preserve the semantic consistency between both modalities. To narrow the modality gap, most recent methods resort to extra object detectors or parsers to obtain the hierarchical representations. In this work, we address this problem by introducing our Multi-Task Hierarchical Convolutional Neural Network (MT-HCN). It is characterized by mining the hierarchical semantic information without the aid of any extra supervisions. Firstly, from the perspective of representing architecture, we leverage the intrinsic hierarchical structure of Convolutional Neural Networks (CNNs) to decompose the representations of both modalities into two semantically complementary levels, i.e., exterior representations and concept representations. The former focuses on discovering the fine-grained low-level associations between both modalities, meanwhile the latter underlines capturing more high-level abstract semantics. Specifically, we present a Self-Supervised Clustering (SSC) loss to preserve more fine-grained semantic clues in exterior representations. It is constituted on the basis of viewing multiple image/text pairs with similar exterior as a category. In addition, a novel harmonious bidirectional triplet ranking (HBTR) loss is proposed, which mitigate the adverse effects brought about by the biased and noisy negative samples. Besides hardest negatives, it also imposes the constraints on the distance between the positive pairs and the centroid of negative pairs. Extensive experimental results on two popular cross-modal retrieval benchmarks demonstrate our proposed MT-HCN can achieve the competitive results compared with the state-of-the-art methods.

Multi-level multilingual semantic alignment for zero-shot cross-lingual transfer learning

Recently, cross-lingual transfer learning has attracted extensive attention from both academia and industry. Previous studies usually focus only on the single-level alignment (e.g., word-level, sentence-level), based on pre-trained language models. However, it leads to suboptimal performance in downstream tasks of the low-resource language due to the missing correlation of hierarchical semantic information (e.g., sentence-to-word, word-to-word). Therefore, in this paper, we propose a novel multi-level alignment framework, which hierarchically learns the semantic correlation between multiple levels by leveraging well-designed alignment training tasks. In addition, we devise an attention-based fusion mechanism (AFM) to infuse semantic information from high levels. Extensive experiments on mainstream cross-lingual tasks (e.g., text classification, paraphrase identification, and named entity recognition) demonstrate the effectiveness of our proposed method, and also show that our model achieves state-of-the-art performance across various benchmarks compared to other strong baselines.

HiSIF-DTA: A Hierarchical Semantic Information Fusion Framework for Drug-Target Affinity Prediction.

Accurately identifying drug-target affinity (DTA) plays a significant role in promoting drug discovery and has attracted increasing attention in recent years. Exploring appropriate protein representation methods and increasing the abundance of protein information is critical in enhancing the accuracy of DTA prediction. Recently, numerous deep learning-based models have been proposed to utilize the sequential or structural features of target proteins. However, these models capture only the low-order semantics that exist in a single protein, while the high-order semantics abundant in biological networks are largely ignored. In this article, we propose HiSIF-DTA'a hierarchical semantic information fusion framework for DTA prediction. In this framework, a hierarchical protein graph is constructed that includes not only contact maps as low-order structural semantics but also protein-rotein interaction (PPI) networks as high-order functional semantics. Particularly, two distinct hierarchical fusion strategies (i.e., Top-down and Bottom-Up) are designed to integrate the different protein semantics, therefore contributing to a richer protein representation. Comprehensive experimental results demonstrate that HiSIF-DTA outperforms current state-of-the-art methods for prediction on the benchmark datasets of the DTA task. Further validation on binary tasks and visualization analysis demonstrates the generalization and interpretation abilities of the proposed method.

Cross-level Feature Aggregation Network for Polyp Segmentation

Accurate segmentation of polyps from colonoscopy images plays a critical role in the diagnosis and cure of colorectal cancer. Although effectiveness has been achieved in the field of polyp segmentation, there are still several challenges. Polyps often have a diversity of size and shape and have no sharp boundary between polyps and their surrounding. To address these challenges, we propose a novel Cross-level Feature Aggregation Network (CFA-Net) for polyp segmentation. Specifically, we first propose a boundary prediction network to generate boundary-aware features, which are incorporated into the segmentation network using a layer-wise strategy. In particular, we design a two-stream structure based segmentation network, to exploit hierarchical semantic information from cross-level features. Furthermore, a Cross-level Feature Fusion (CFF) module is proposed to integrate the adjacent features from different levels, which can characterize the cross-level and multi-scale information to handle scale variations of polyps. Further, a Boundary Aggregated Module (BAM) is proposed to incorporate boundary information into the segmentation network, which enhances these hierarchical features to generate finer segmentation maps. Quantitative and qualitative experiments on five public datasets demonstrate the effectiveness of our CFA-Net against other state-of-the-art polyp segmentation methods. The source code and segmentation maps will be released at https://github.com/taozh2017/CFANet.

End-to-end Pre-training with Hierarchical Matching and Momentum Contrast for Text-Video Retrieval.

Lately, video-language pre-training and text-video retrieval have attracted significant attention with the explosion of multimedia data on the Internet. However, existing approaches for video-language pre-training typically limit the exploitation of the hierarchical semantic information in videos, such as frame semantic information and global video semantic information. In this work, we present an end-to-end pre-training network with Hierarchical Matching and Momentum Contrast named HMMC. The key idea is to explore the hierarchical semantic information in videos via multilevel semantic matching between videos and texts. This design is motivated by the observation that if a video semantically matches a text (can be a title, tag or caption), the frames in this video usually have semantic connections with the text and show higher similarity than frames in other videos. Hierarchical matching is mainly realized by two proxy tasks: Video-Text Matching (VTM) and Frame-Text Matching (FTM). Another proxy task: Frame Adjacency Matching (FAM) is proposed to enhance the single visual modality representations while training from scratch. Furthermore, momentum contrast framework was introduced into HMMC to form a multimodal momentum contrast framework, enabling HMMC to incorporate more negative samples for contrastive learning which contributes to the generalization of representations. We also collected a large-scale Chinese video-language dataset (over 763k unique videos) named CHVTT to explore the multilevel semantic connections between videos and texts. Experimental results on two major Text-video retrieval benchmark datasets demonstrate the advantages of our methods. We release our code at https://github.com/cheetah003/HMMC.

Context vector-based visual mapless navigation in indoor using hierarchical semantic information and meta-learning

Visual mapless navigation (VMN), modeling a direct mapping between sensory inputs and agent actions, aims to navigate from a stochastic origin location to a prescribed goal in an unseen scene. A fundamental yet challenging issue in visual mapless navigation is generalizing to a new scene. Furthermore, it is of pivotal concern to design a method to make effective policy learning. To address these issues, we introduce a novel visual mapless navigation model, which integrates hierarchical semantic information represented by context vector with meta-learning to improve the generalization performance gap between known and unknown environments. Extensive experimental results on AI2-THOR benchmark dataset demonstrate that our model significantly outperforms the state-of-the-art model by 15.79% for the SPL and by 23.83% for the success rate. In addition, the exploration rate experiment shows that our model can effectively improve the invalid exploration behavior of the agent and accelerate the convergence speed of the model. Our implementation code and data can be viewed on https://github.com/zhiyu-tech/WHU-CVVMN.

DDNet: 3D densely connected convolutional networks with feature pyramids for nasopharyngeal carcinoma segmentation

Radiation therapy is the standard treatment for early stage Nasopharyngeal cancer (NPC). Thus, accurate delineation of target volumes at risk in NPC is important. While manual delineation is time-consuming and labour-intensive process and also leads to significant inter- and intra-practitioner variability. Thus, computer-aided segmentation algorithm is required. However, segmentation task is not trivial due to large variations (e.g., shape and size) of nasopharynx structure across subjects. Moreover, extreme foreground and background class imbalance in NPC segmentation remains challenge. In this paper, we propose a threedimensional densely connected convolutional neural network with multi-scale feature pyramids for NPC segmentation. We adapt the densely connected convolutional block into a new structure via adding feature pyramids. The concatenated pyramid feature carries multi-scale and hierarchical semantic information which is effective for segmenting different size of tumors and perceiving hierarchical context information. To address the foreground and background imbalance problem, we propose an enhanced version of focal loss. It prevents the large number of negative voxels far from boundaries from overwhelming the segmentation algorithm. We validated the proposed method on 120 clinical subjects. Experimental results demonstrate that our approach out-performed state-of-the-art methods and human experts.

Traditional Chinese medicine symptom normalization approach leveraging hierarchical semantic information and text matching with attention mechanism

Traditional Chinese medicine (TCM) symptom normalization is difficult because the challenges of the symptoms having different literal descriptions, one-to-many symptom descriptions and different symptoms sharing a similar literal description. We propose a novel two-step approach utilizing hierarchical semantic information that represents the functional characteristics of symptoms and develop a text matching model that integrates hierarchical semantic information with an attention mechanism to solve these problems. In this study, we constructed a symptom normalization dataset and a TCM normalization symptom dictionary containing normalization symptom words, and assigned symptoms into 24 classes of functional characteristics. First, we built a multi-label text classifier to isolate the hierarchical semantic information from each symptom description and count the corresponding normalization symptoms and filter the candidate set. Then we designed a text matching model of mixed multi-granularity language features with an attention mechanism that utilizes the hierarchical semantic information to calculate the matching score between the symptom description and the normalization symptom words. We compared our approach with other baselines on real-world data. Our approach gives the best performance with a Hit@ 1, 3, and 10 of 0.821, 0.953, and 0.993, respectively, and a MeanRank of 1.596, thus outperforming significantly regarding the symptom normalization task. We developed an approach for the TCM symptom normalization task and demonstrated its superior performance compared with other baselines, indicating the promise of this research direction.

Hierarchical Semantic Propagation for Object Detection in Remote Sensing Imagery

Object detection in remote sensing imagery is a critical yet challenging task in the field of computer vision due to the bird’s-eye-view perspective. Although existing object detection approaches in remote sensing imagery have achieved great advances through the utilization of deep features or rotation proposals, but they give insufficient consideration to multilevel semantic information and its propagation for guiding the learning process. Accordingly, in this article, we propose a hierarchical semantic propagation (HSP) framework to boost object detection performance in remote sensing imagery, which is better able to propagate hierarchical semantic information among different components in a unified network. Given a remote sensing image as input, the HSP framework can detect instances of semantic objects belonging to certain categories in an end-to-end way. First, the multiscale representation is captured by a basic feature pyramid network, which can hierarchically combine spatial attention details and the global semantic structure in order to learn more discriminative visual features. Second, the soft-segmentation prediction is used as an auxiliary objective in the intermediate layer of our HSP; its output instance-aware semantic information can be propagated to suppress noisy background information and thereby guide the proposal generation in the region proposal network. By further propagating this hierarchical semantic information into the region of interest module, we can then predict the object category information and the corresponding horizontal and oriented bounding boxes. Comprehensive evaluations on three benchmark data sets demonstrate the superiority of our HSP to the existing state-of-the-art methods for object detection in remote sensing imagery.

Named-Entity Recognition in Sports Field Based on a Character-Level Graph Convolutional Network

Traditional methods for identifying naming ignore the correlation between named entities and lose hierarchical structural information between the named entities in a given text. Although traditional named-entity methods are effective for conventional datasets that have simple structures, they are not as effective for sports texts. This paper proposes a Chinese sports text named-entity recognition method based on a character graph convolutional neural network (Char GCN) with a self-attention mechanism model. In this method, each Chinese character in the sports text is regarded as a node. The edge between the nodes is constructed using a similar character position and the character feature of the named-entity in the sports text. The internal structural information of the entity is extracted using a character map convolutional neural network. The hierarchical semantic information of the sports text is captured by the self-attention model to enhance the relationship between the named entities and capture the relevance and dependency between the characters. The conditional random fields classification function can accurately identify the named entities in the Chinese sports text. The results conducted on four datasets demonstrate that the proposed method improves the F-Score values significantly to 92.51%, 91.91%, 93.98%, and 95.01%, respectively, in comparison to the traditional naming methods.

A Hybrid Markov Random Field Model With Multi-Granularity Information for Semantic Segmentation of Remote Sensing Imagery

High-spatial-resolution (HSR) remote sensing images usually contain rich hierarchical semantic information. However, many methods fail to solve the segmentation misclassification problems for HSR images due to just considering one layer of granularity information, such as the pixel granularity layer or the object granularity layer. This paper presents a hybrid Markov random field model for the semantic segmentation of HSR images by paying closer attention to capture the multi-granularity information. In this model, a probability graph with a multilayer structure is first built to represent different granularities information. Then, a hybrid label field is developed to model the multi-granularity classes with the form of a vector, and a new joint distribution is designed to capture the isotropic spatial interactions within each layer of the hybrid label field and the anisotropic spatial interactions between different layers. A generative probabilistic inference is proposed to realize the synergy between the multi-granularity information and the hybrid label interactions by iteratively updating the likelihood function and the joint probability of the hybrid label. The final semantic segmentation result can be achieved when the probabilistic inference converges. Experimental results over different HSR remote sensing images show that the proposed method can achieve more accurate segmentation than other state-of-the-art methods.

Semantic Segmentation of Remote Sensing Imagery Using an Object-Based Markov Random Field Model With Auxiliary Label Fields

The Markov random field (MRF) model has attracted great attention in the field of image segmentation. However, most MRF-based methods fail to resolve segmentation misclassification problems for high spatial resolution remote sensing images due to insufficiently using the hierarchical semantic information. In order to solve such a problem, this paper proposes an object-based MRF model with auxiliary label fields that can capture more macro and detailed information and apply it to the semantic segmentation of high spatial resolution remote sensing images. Specifically, apart from the label field, two auxiliary label fields are first introduced into the proposed model for interpreting remote sensing images from different perspectives, which are implemented by setting a different number of auxiliary classes. Then, the multilevel logistic model is used to describe the interactions within each label field, and a conditional probability distribution is developed to model the interactions between label fields. A net context structure is established among them to model the interactions of classes within and between label fields. A principled probabilistic inference is suggested to solve the proposed model by iteratively renewing the label field and auxiliary label fields, in which different information of auxiliary label fields can be integrated into the label field during iterations. Experiments on different remote sensing images demonstrate that our model produces more accurate segmentation than the state-of-the-art MRF-based methods. If some prior information is added, the proposed model can produce accurate results even in complex areas.

Multilayer semantic segmentation of remote-sensing imagery using a hybrid object-based Markov random field model

ABSTRACTHigh spatial resolution (HR) remote-sensing image usually contains hierarchical semantic information. Many supervised methods have been developed to interpret this information through data training. In this article, without data training, a hybrid object-based Markov random field (HOMRF) model is proposed for multilayer semantic segmentation of remote-sensing images. In this method, label fields of different semantic layers are defined on the same region adjacency graph (RAG) of a given image, and a hybrid framework is suggested to capture and utilize the interactions within and between semantic layers by label fields. Namely a new transition probability matrix is introduced into the energy functions of label fields for describing the semantic context between layers, and the multilevel logistic model is employed to describe the interactions within the same layer. A principled probabilistic inference is developed to determine the optimal solution of the proposed method by iteratively updating each label field until convergence. The computational complexity of the proposed model is , where is the number of classes in all of the layers, is the number of sites in the probability graph of the MRF model, and is the number of iterations. Experimental results from various remote-sensing images demonstrate that the proposed method can produce higher segmentation accuracy than state-of-the-art MRF-based methods.

Automatic hierarchical mid-surface abstraction of thin-walled model based on rib decomposition

Model simplification is imperative in the process of computer aided design (CAD) and computer aided engineering (CAE) integration. Mid-surface abstraction is the most effective method to simplify the thin-walled models. Many previous research efforts have been focused on the mid-surface abstraction, including the model decomposition based methods, Medial Axis Transform (MAT) based methods and Chordal Axis Transform (CAT) based methods. However, complex thin-walled models cannot be handled well due to the fact that there are some problems including low geometrical precision, poor topological structure, etc., in the above resultant mid-surface models. Especially, these methods are hard to be reused to generate the mid-surface model efficiently. Therefore, a hierarchical semantic mid-surface abstraction method is proposed for the thin-walled model based on rib feature decomposition in this paper. Firstly, a new hierarchical semantic structure is defined and applied on both the thin-walled models and mid-surface models. After that, the model decomposition is conducted based on the identified rib features and the hierarchical semantic information is obtained at the same time. Then, the offset operation and discretization based methods are used to obtain the mid-surface patch for each sub region with different semantic structure respectively. Finally, the mid-surface model with hierarchical semantic information is generated by stitching all discrete patches. Moreover, the above model can be reused to facilitate the rapid mid-surface abstraction of the changed model based on its hierarchical semantic structure. Several examples are given to demonstrate the outperformance of the proposed method.

Hierarchical semantic information modeling and ontology for bird ecology

Birds have become an increasing concern for ecological preservation and safety. This paper proposes hierarchical architecture of semantic sensing information for bird acoustic data representation in bird ecological environment. This architecture provides various real-time sensing data such as bird calls using acoustic sensors in sensor networks. In this paper, we implement an ontology structure of hierarchical semantic information representation in bird's ecological environment. Information of this architecture supports to recognize bird calls, identify birds, classify species, and to track a bird behavior in bird ecological environment. All of this would indicate that we suggest relationship between phenomenon data to service/semantic information in bird ecology.