Hierarchical Encoding Research Articles

Digital-Reported Outcome from Medical Notes of Schizophrenia and Bipolar Patients Using Hierarchical BERT

Patient-reported (PRO) and clinician-reported (CRO) outcomes are assessment instruments that are completed by patients and trained healthcare professionals, respectively. A PRO is a report of the direct experience of the patient with a given disease condition. A CRO is an assessment of the condition of the patient by the healthcare provider. PROs may not be accessible to all patients, especially those suffering from severe disease conditions. CROs are time-consuming and therefore administered infrequently. In the present study, we introduce a new form of assessment, the digital-reported outcome (DRO), which is automatically derived from the medical notes of the patient. DROs have a low overhead and can be generated at each patient’s visit to complement other outcome-assessment instruments and enhance clinical decision support by identifying at-risk patients. In this study, a DRO is developed to evaluate the functional impairment in the daily activities of two cohorts of patients suffering from bipolar disorder and schizophrenia. The input of the DRO is a single medical note from the electronic medical record of the patient. This note is submitted to a hierarchical bidirectional encoder representations from transformers (BERT) model. First, a sentence-level embedding is produced for each sentence in the note using a token-level attention mechanism. Second, an embedding for the entire note is constructed using a sentence-level attention mechanism. Third, the final embedding is classified using a feed-forward neural network. The model is trained to classify patients into moderate or severe functioning impairment levels according to the general assessment of functioning (GAF) scale, a CRO instrument for the assessment of the impact of mental illness on the daily activities of the patient. The DRO is validated using medical notes that were labeled by multiple healthcare providers from different healthcare institutions. The results indicate that a general DRO is able to classify patients from the two cohorts according to the two functioning impairment levels (severe versus moderate) prior to the onset of disease with an AUC of 76%. Disease-specific DROs are only applicable after the onset of the disease and produced AUCs of nearly 85%. The methodology introduced in the present paper is practical and can support the automated monitoring of the severity of the functioning impairment of bipolar and schizophrenia patients. Extending the proposed DRO to other psychiatric conditions and types of impairments is the subject of ongoing research work.

Modified Hierarchical-Attention Network model for legal judgment predictions

The impact of Artificial Intelligence in Legal Research has reached a high level in simulating human thought processes. Case Pendency is a long-lasting problem in many countries. The judicial system has to be more competent and reliable to provide justice on time for any developing country. Litigants and attorneys devote more time and effort to trial case preparation in the courtroom. The task of decision prediction is to automatically forecast the type of charge, law article, and term of punishment. Most of the earlier works for verdict prediction focused to work on civil law jurisdictions. Some of the challenges in the task are case facts are highly unstructured lengthy documents with a lack of annotations and mainly used machine learning techniques. While most research works ignore the information loss at the encoding stage, our proposed MHAN overcomes the above issue and long-range dependency problem using the attention model over hierarchical encoders with three tiers namely Sentence encoder, word encoder, and character encoder. To avoid information loss, a brand-new judgment prediction framework called MHAN is developed in this study effort. It is built on a modified Hierarchical-Attention network and a specially designed domain-specific word embedding model. Additionally, it emphasizes the feature extraction phase by joining features obtained using MHAN with an improved cosine similarity feature. Finally, a hybrid Self Improved RNN is employed to provide the projected results. Furthermore, the proposed model is trained on 10 types of real-time criminal cases from the Madras High Court of India and Supreme Court of India. It has outperformed prior methods in terms of verdict prediction. By applying different variations of the deep learning model and ablation tests, the proposed model achieves consistent results over baseline models.

Reconstructing controllable faces from brain activity with hierarchical multiview representations

Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview (i.e., visual stimuli and evoked fMRI) latent representations.

Next Point-of-Interest Recommendation Based on Joint Mining of Spatial–Temporal and Semantic Sequential Patterns

With the widespread use of the location-based social networks (LBSNs), the next point-of-interest (POI) recommendation has become an essential service, which aims to understand the user’s check-in behavior at the current moment by analyzing and mining the correlations between the user’s check-in behaviors within his/her historical trajectory and then recommending the POI that the user is most likely to visit at the next time step. However, the user’s check-in trajectory presents extremely irregular sequential patterns, such as spatial–temporal patterns, semantic patterns, etc. Intuitively, the user’s visiting behavior is often accompanied by a certain purpose, which makes the check-in data in LBSNs often have rich semantic activity characteristics. However, existing research mainly focuses on exploring the spatial–temporal sequential patterns and lacks the mining of semantic information within the trajectory, so it is difficult to capture the user’s visiting intention. In this paper, we propose a self-attention- and multi-task-based method, called MSAN, to explore spatial–temporal and semantic sequential patterns simultaneously. Specifically, the MSAN proposes to mine the user’s visiting intention from his/her semantic sequence and uses the user’s visiting intention prediction task as the auxiliary task of the next POI recommendation task. The user’s visiting intention prediction uses hierarchical POI category attributes to describe the user’s visiting intention and designs a hierarchical semantic encoder (HSE) to encode the hierarchical intention features. Moreover, a self-attention-based hierarchical intention-aware module (HIAM) is proposed to mine temporal and hierarchical intention features. The next POI recommendation uses the self-attention-based spatial–temporal-aware module (STAM) to mine the spatial–temporal sequential patterns within the user’s check-in trajectory and fuses this with the hierarchical intention patterns to generate the next POI list. Experiments based on two real datasets verified the effectiveness of the model.

One-pot ternary sequential reactions for photopatterned gradient multimaterials

One-pot ternary sequential reactions for photopatterned gradient multimaterials

Hierarchical graph attention network for temporal knowledge graph reasoning

Temporal knowledge graphs (TKGs) reasoning has attracted increasing research interest in recent years. However, most of the existing TKGs reasoning models aim to learn a dynamic entity representation by binding timestamps information with the entities, neglecting to learn adaptive entity representation that is valuable to the query from relevant historical facts. To this end, we propose a Hierarchical Graph Attention neTwork (HGAT) for the TKGs reasoning task. Specifically, we design a hierarchical neighbor encoder to model the time-oriented and task-oriented roles of the entities. The time-aware mechanism is developed in the first layer to differentiate the contributions of query-relevant historical facts at different timestamps to the query. The designed relation-aware attention is used in the second layer to discern the contributions of the structural neighbors of an entity. Through this hierarchical encoder, our model can absorb valuable knowledge effectively from the relevant historical facts, and thus learn more expressive adaptive entity representation for the query. Finally, we evaluate our model performance on four TKGs datasets and justify its superiority against vaerious state-of-the-art baselines.

Structured visuospatial representations revealed through serial reproduction.

Working memory is a reconstructive process that requires integrating multiple hierarchical representations of objects. This hierarchical reconstruction allows us to overcome perceptual uncertainty and limited cognitive capacity but yields systematic biases in working memory as individual items are influenced by the ensemble statistics of the scene, or of their particular group. Given the importance of the hierarchical encoding of a display, we aim to characterize what structures people use to encode visual scenes using a nonparametric data-driven approach. In Experiment 1, we examine visuospatial memory for locations by asking participants to recall the locations of objects in a serial reproduction task. We show that people report items in a more compact structure than they initially were and organize them into clustered spatial groups. In Experiment 2, we explicitly introduce discrete color groups, allowing us to test whether the color feature governs the spatial grouping. We find that the spatial structures were color-contingent. By analyzing color groups, we circumvent the grouping uncertainty in Experiment 1 and further reveal that people compress color groups into collinear structures with similar orientations and equidistant spacing. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Hierarchical Residual Encoding for Multiresolution Time Series Compression

Data compression is a key technique for reducing the cost of data transfer from storage to compute nodes. Increasingly, modern data scales necessitate lossy compression techniques, where exactness is sacrificed for a smaller compressed representation. One challenge in lossy compression is that different applications may have different accuracy demands. Today's compression techniques struggle in this setting either forcing the user to compress at the strictest accuracy demand, or to re-encode the data at multiple resolutions. This paper proposes a simple, but effective multiresolution compression algorithm for time series data, where a single encoding can effectively be decompressed at multiple output resolutions. There are a number of benefits over current state-of-the-art techniques for time series compression. (1) The storage footprint of this encoding is smaller than re-encoding the data at multiple resolutions. (2) Similarly, the compression latency is generally smaller than re-encoding at multiple resolutions. (3) Finally, the decompression latency of our encoding is significantly faster than single encodings at the strictest accuracy demand.

Hierarchical Color Encoding for Progressive Image Transmission in Underwater Environments

This paper presents the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Progressive Hierarchical Image Encoding</i> (PHIE) approach aimed at progressively encoding underwater images and, thus, helping in transmitting them in unreliable, low bandwidth, subaquatic communication channels. The proposal splits an image into several, very small, fragments that can be transmitted separately. The receiver can reconstruct the image after receiving the first fragment and improve the reconstruction quality with each received fragment up to a perfect, lossless, reconstruction. An experimental comparative study using real underwater imagery and showing the benefits and drawbacks of the proposed approach is also presented.

MISSFormer: An Effective Transformer for 2D Medical Image Segmentation.

Transformer-based methods are recently popular in vision tasks because of their capability to model global dependencies alone. However, it limits the performance of networks due to the lack of modeling local context and global-local correlations of multi-scale features. In this paper, we present MISSFormer, a Medical Image Segmentation tranSFormer. MISSFormer is a hierarchical encoder-decoder network with two appealing designs: 1) a feed-forward network in transformer block of U-shaped encoder-decoder structure is redesigned, ReMix-FFN, which explore global dependencies and local context for better feature discrimination by re-integrating the local context and global dependencies; 2) a ReMixed Transformer Context Bridge is proposed to extract the correlations of global dependencies and local context in multi-scale features generated by our hierarchical transformer encoder. The MISSFormer shows a solid capacity to capture more discriminative dependencies and context in medical image segmentation. The experiments on multi-organ, cardiac segmentation and retinal vessel segmentation tasks demonstrate the superiority, effectiveness and robustness of our MISSFormer. Specifically, the experimental results of MISSFormer trained from scratch even outperform state-of-the-art methods pre-trained on ImageNet, and the core designs can be generalized to other visual segmentation tasks. The code has been released on Github: https://github.com/ZhifangDeng/MISSFormer.

Distributed and hierarchical neural encoding of multidimensional biological motion attributes in the human brain.

The human visual system can efficiently extract distinct physical, biological, and social attributes (e.g. facing direction, gender, and emotional state) from biological motion (BM), but how these attributes are encoded in the brain remains largely unknown. In the current study, we used functional magnetic resonance imaging to investigate this issue when participants viewed multidimensional BM stimuli. Using multiple regression representational similarity analysis, we identified distributed brain areas, respectively, related to the processing of facing direction, gender, and emotional state conveyed by BM. These brain areas are governed by a hierarchical structure in which the respective neural encoding of facing direction, gender, and emotional state is modulated by each other in descending order. We further revealed that a portion of the brain areas identified in representational similarity analysis was specific to the neural encoding of each attribute and correlated with the corresponding behavioral results. These findings unravel the brain networks for encoding BM attributes in consideration of their interactions, and highlight that the processing of multidimensional BM attributes is recurrently interactive.

Colorectal polyp segmentation method based on fusion of transformer and cross-level phase awareness

In order to address the issues of spatial induction bias and lack of effective representation of global contextual information in colon polyp image segmentation, which lead to the loss of edge details and mis-segmentation of lesion areas, a colon polyp segmentation method that combines Transformer and cross-level phase-awareness is proposed. The method started from the perspective of global feature transformation, and used a hierarchical Transformer encoder to extract semantic information and spatial details of lesion areas layer by layer. Secondly, a phase-aware fusion module (PAFM) was designed to capture cross-level interaction information and effectively aggregate multi-scale contextual information. Thirdly, a position oriented functional module (POF) was designed to effectively integrate global and local feature information, fill in semantic gaps, and suppress background noise. Fourthly, a residual axis reverse attention module (RA-IA) was used to improve the network's ability to recognize edge pixels. The proposed method was experimentally tested on public datasets CVC-ClinicDB, Kvasir, CVC-ColonDB, and EITS, with Dice similarity coefficients of 94.04%, 92.04%, 80.78%, and 76.80%, respectively, and mean intersection over union of 89.31%, 86.81%, 73.55%, and 69.10%, respectively. The simulation experimental results show that the proposed method can effectively segment colon polyp images, providing a new window for the diagnosis of colon polyps.

Hierarchical Associative Encoding and Decoding for Bottom-Up Human Pose Estimation

Bottom-up human pose estimation decouples computational complexity from the number of people but requires additional operations to match the detected keypoints to each human instance. Existing approaches treat all keypoints equally while ignoring the relationships among keypoints, which in turn limit the performance ceilings. In this work, we propose a hierarchical associative encoding and decoding framework for bottom-up human pose estimation by introducing additional prior knowledge. Specifically, in addition to keypoint-level and instance-level associations, we further divide keypoints into groups and explore group-level associations. This way, prior knowledge is incorporated to determine the keypoint groups for better associative encoding. To deal with complex poses, we introduce a focal pulling loss to focus more on the hard-to-associate keypoints. Moreover, instead of using a pre-defined order for keypoint grouping, we propose a progressive associative decoding method to dynamically determine the order of keypoints for grouping, which helps reduce isolated keypoints. Experimental results on the MS-COCO, CrowdPose and MPII datasets show superior performance of our proposed associative encoding and decoding algorithms. More importantly, we prove, through validation, that hierarchical associative encoding and decoding can be used as a plug-n-play module for performance improvement regardless of backbone architecture. Our source code and pretrained models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ducongju/HAE</uri> .

Structure and Sequence Aligned Code Summarization with Prefix and Suffix Balanced Strategy

Source code summarization focuses on generating qualified natural language descriptions of a code snippet (e.g., functionality, usage and version). In an actual development environment, descriptions of the code are missing or not consistent with the code due to human factors, which makes it difficult for developers to comprehend and conduct subsequent maintenance. Some existing methods generate summaries from the sequence information of code without considering the structural information. Recently, researchers have adopted the Graph Neural Networks (GNNs) to capture the structural information with modified Abstract Syntax Trees (ASTs) to comprehensively represent a source code, but the alignment method of the two information encoder is hard to decide. In this paper, we propose a source code summarization model named SSCS, a unified transformer-based encoder-decoder architecture, for capturing structural and sequence information. SSCS is designed upon a structure-induced transformer with three main novel improvements. SSCS captures the structural information in a multi-scale aspect with an adapted fusion strategy and adopts a hierarchical encoding strategy to capture the textual information from the perspective of the document. Moreover, SSCS utilizes a bidirectional decoder which generates a summary from opposite direction to balance the generation performance between prefix and suffix. We conduct experiments on two public Java and Python datasets to evaluate our method and the result show that SSCS outperforms the state-of-art code summarization methods.

FE-GAN: Fast and Efficient Underwater Image Enhancement Model Based on Conditional GAN

The processing of underwater images can vastly ease the difficulty of underwater robots’ tasks and promote ocean exploration development. This paper proposes a fast and efficient underwater image enhancement model based on conditional GAN with good generalization ability using aggregation strategies and concatenate operations to take full advantage of the limited hierarchical features. A sequential network can avoid frequently visiting additional nodes, which is beneficial for speeding up inference and reducing memory consumption. Through the structural re-parameterization approach, we design a dual residual block (DRB) and accordingly construct a hierarchical attention encoder (HAE), which can extract sufficient feature and texture information from different levels of an image, and with 11.52% promotion in GFLOPs. Extensive experiments were carried out on real and artificially synthesized benchmark underwater image datasets, and qualitative and quantitative comparisons with state-of-the-art methods were implemented. The results show that our model produces better images, and has good generalization ability and real-time performance, which is more conducive to the practical application of underwater robot tasks.

Energy-Efficient Multi-UAVs Cooperative Trajectory Optimization for Communication Coverage: An MADRL Approach

Unmanned Aerial Vehicles (UAVs) can be deployed as aerial wireless base stations which dynamically cover the wireless communication networks for Ground Users (GUs). The most challenging problem is how to control multi-UAVs to achieve on-demand coverage of wireless communication networks while maintaining connectivity among them. In this paper, the cooperative trajectory optimization of UAVs is studied to maximize the communication efficiency in the dynamic deployment of UAVs for emergency communication scenarios. We transform the problem into a Markov game problem and propose a distributed trajectory optimization algorithm, Double-Stream Attention multi-agent Actor-Critic (DSAAC), based on Multi-Agent Deep Reinforcement Learning (MADRL). The throughput, safety distance, and power consumption of UAVs are comprehensively taken into account for designing a practical reward function. For complex emergency communication scenarios, we design a double data stream network structure that provides a capacity for the Actor network to process state changes. Thus, UAVs can sense the movement trends of the GUs as well as other UAVs. To establish effective cooperation strategies for UAVs, we develop a hierarchical multi-head attention encoder in the Critic network. This encoder can reduce the redundant information through the attention mechanism, which resolves the problem of the curse of dimensionality as the number of both UAVs and GUs increases. We construct a simulation environment for emergency networks with multi-UAVs and compare the effects of the different numbers of GUs and UAVs on algorithms. The DSAAC algorithm improves communication efficiency by 56.7%, throughput by 71.2%, energy saving by 19.8%, and reduces the number of crashes by 57.7%.

Generating Relevant and Informative Questions for Open-Domain Conversations

Recent research has highlighted the importance of mixed-initiative interactions in conversational search. To enable mixed-initiative interactions, information retrieval systems should be able to ask diverse questions, such as information-seeking, clarification, and open-ended ones. question generation (QG) of open-domain conversational systems aims at enhancing the interactiveness and persistence of human-machine interactions. The task is challenging because of the sparsity of question generation (QG)-specific data in conversations. Current work is limited to single-turn interaction scenarios. We propose a context-enhanced neural question generation (CNQG) model that leverages the conversational context to predict question content and pattern, then perform question decoding. A hierarchical encoder framework is employed to obtain the discourse-level context representation. Based on this, we propose Review and Transit mechanisms to respectively select contextual keywords and predict new topic words to further construct the question content. Conversational context and the predicted question content are used to produce the question pattern, which in turn guides the question decoding process implemented by a recurrent decoder with a joint attention mechanism. To fully utilize the limited QG-specific data to train our question generator, we perform multi-task learning with three auxiliary training objectives, i.e., question pattern prediction, Review , and Transit mechanisms. The required additional labeled data is obtained in a self-supervised way. We also design a weight decaying strategy to adjust the influences of various auxiliary learning tasks. To the best of our acknowledge, we are the first to extend the application of QG to the multi-turn open-domain conversational scenario. Extensive experimental results demonstrate the effectiveness of our proposal and its main components on generating relevant and informative questions, with robust performance for contexts with various lengths.

HierTTS: Expressive End-to-End Text-to-Waveform Using a Multi-Scale Hierarchical Variational Auto-Encoder

End-to-end text-to-speech (TTS) models that directly generate waveforms from text are gaining popularity. However, existing end-to-end models are still not natural enough in their prosodic expressiveness. Additionally, previous studies on improving the expressiveness of TTS have mainly focused on acoustic models. There is a lack of research on enhancing expressiveness in an end-to-end framework. Therefore, we propose HierTTS, a highly expressive end-to-end text-to-waveform generation model. It deeply couples the hierarchical properties of speech with hierarchical variational auto-encoders and models multi-scale latent variables, at the frame, phone, subword, word, and sentence levels. The hierarchical encoder encodes the speech signal from fine-grained features into coarse-grained latent variables. In contrast, the hierarchical decoder generates fine-grained features conditioned on the coarse-grained latent variables. We propose a staged KL-weighted annealing strategy to prevent hierarchical posterior collapse. Furthermore, we employ a hierarchical text encoder to extract linguistic information at different levels and act on both the encoder and the decoder. Experiments show that our model performs closer to natural speech in prosody expressiveness and has better generative diversity.

Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation.

The optical flow guidance strategy is ideal for obtaining motion information of objects in the video. It is widely utilized in video segmentation tasks. However, existing optical flow-based methods have a significant dependency on optical flow, which results in poor performance when the optical flow estimation fails for a particular scene. The temporal consistency provided by the optical flow could be effectively supplemented by modeling in a structural form. This paper proposes a new hierarchical graph neural network (GNN) architecture, dubbed hierarchical graph pattern understanding (HGPU), for zero-shot video object segmentation (ZS-VOS). Inspired by the strong ability of GNNs in capturing structural relations, HGPU innovatively leverages motion cues (i.e., optical flow) to enhance the high-order representations from the neighbors of target frames. Specifically, a hierarchical graph pattern encoder with message aggregation is introduced to acquire different levels of motion and appearance features in a sequential manner. Furthermore, a decoder is designed for hierarchically parsing and understanding the transformed multi-modal contexts to achieve more accurate and robust results. HGPU achieves state-of-the-art performance on four publicly available benchmarks (DAVIS-16, YouTube-Objects, Long-Videos and DAVIS-17). Code and pre-trained model can be found at https://github.com/NUST-Machine-Intelligence-Laboratory/HGPU.

Progressive Downsampling Transformer With Convolution-Based Decoder and Its Application in Gear Pitting Measurement

In Transformer for semantic segmentation, patch embedding usually has only one convolutional layer with a large stride, leading to the decrease of feature extraction capability. Additionally, the complex decoder results in high computation cost. To address above-mentioned two issues, we put forward a progressive downsampling Transformer with convolution-based decoder (PDCDT), which is a simple, efficient yet powerful framework. Specifically, progressive downsampling layers for patch embedding are designed to refine the extracted features and reduce information loss at each stage of the hierarchical Transformer encoder. Meanwhile, a simple decoder based on a convolution (conv) module is proposed for aggregating the characteristic information from multiscale output layers of the encoder, and it can realize dimensional transformation and information interaction with fewer parameters than the decoders used in the existing Transformers. Extensive experiments show that PDCDT achieves competitive results on ADE20K (47.9% mIoU) and Cityscapes (82.6% mIoU). Finally, PDCDT is applied to gear pitting measurement in gear contact fatigue test, and the comparative results indicate that PDCDT can improve the accuracy of pitting detection.