Embedding Representation Research Articles

A heterogeneous 3-D stacked PIM accelerator for GCN-based recommender systems

AbstractModern recommendation systems integrate graph convolution neural networks (GCN) for enhancing embedding representation. Compared with widely deployed neural network-based models, the extra message propagation layer of GCN-based recommendation is featured with extensive computations and irregular memory access. However, architecture designs for prevailing deep neural network recommendation models assume simple pooling in the embedding layer. ReRAM-based GCN accelerators are specialized for graph-related operations. However, they are designed for general graphs, while GCN-based recommendation models mainly operate on the user-item graph. In this paper, we proposed a resistive random accessed memory (ReRAM) based processing-in-memory (PIM) accelerator, ReGCNR, for GCN-based recommendation. ReGCNR is featured with three key innovations. First, we exploit the 3-dimensional (3-D) stacked heterogeneous ReRAM to fit with the large-size embedding table and user-item graph. Then, we propose a joint degree mapping schema that maximizes the efficiency of the execution pipeline. After that, ReGCNR assembles a well-coordinated pipeline and hardware scheduling design to boost overall system performance. Results show that ReGCNR outperforms GPU by 69.83$$\times$$ × and 56.67$$\times$$ × in terms of average speedup and energy saving, respectively. In addition, ReGCNR outperforms state-of-the-art ReRAM-based solutions by 11.13$$\times$$ × speedups and 7.22$$\times$$ × energy savings on average.

Attribute Graph Embedding Based on Multi-Order Adjacency Views and Attention Mechanisms

Graph embedding plays an important role in the analysis and study of typical non-Euclidean data, such as graphs. Graph embedding aims to transform complex graph structures into vector representations for further machine learning or data mining tasks. It helps capture relationships and similarities between nodes, providing better representations for various tasks on graphs. Different orders of neighbors have different impacts on the generation of node embedding vectors. Therefore, this paper proposes a multi-order adjacency view encoder to fuse the feature information of neighbors at different orders. We generate different node views for different orders of neighbor information, consider different orders of neighbor information through different views, and then use attention mechanisms to integrate node embeddings from different views. Finally, we evaluate the effectiveness of our model through downstream tasks on the graph. Experimental results demonstrate that our model achieves improvements in attributed graph clustering and link prediction tasks compared to existing methods, indicating that the generated embedding representations have higher expressiveness.

A domain-label-guided translation model for molecular optimization

Molecular optimization, which aims to improve molecular properties by modifying complex molecular structures, is a crucial and challenging task in drug discovery. In recent years, translation models provide a promising way to transform low-property molecules to high-property molecules, which enables molecular optimization to achieve remarkable progress. However, most existing models require matched molecular pairs, which are prone to be limited by the datasets. Although some models do not require matched molecular pairs, their performance is usually sacrificed due to the lack of useful supervising information. To address this issue, a domain-label-guided translation model is proposed in this paper, namely DLTM. In the model, the domain label information of molecules is exploited as a control condition to obtain different embedding representations, enabling the model to generate diverse molecules. Besides, the model adopts a classifier network to identify the property categories of transformed molecules, guiding the model to generate molecules with desired properties. The performance of DLTM is verified on two optimization tasks, namely the quantitative estimation of drug-likeness and penalized logP. Experimental results show that the proposed DLTM is superior to the compared baseline models.

Distributed Detection of Large-Scale Internet of Things Botnets Based on Graph Partitioning

With the rapid growth of IoT devices, the threat of botnets is becoming increasingly worrying. There are more and more intelligent detection solutions for botnets that have been proposed with the development of artificial intelligence. However, due to the current lack of computing power in IoT devices, these intelligent methods often cannot be well-applied to IoT devices. Based on the above situation, this paper proposes a distributed botnet detection method based on graph partitioning, efficiently detecting botnets using graph convolutional networks. In order to alleviate the wide range of IoT environments and the limited computing power of IoT devices, the algorithm named METIS is used to divide the network traffic structure graph into small graphs. To ensure robust information flow between nodes while preventing gradient explosion, diagonal enhancement is applied to refine the embedding representations at each layer, facilitating accurate botnet attack detection. Through comparative analysis with GATv2, GraphSAGE, and GCN across the C2, P2P, and Chord datasets, our method demonstrates superior performance in both accuracy and F1 score metrics. Moreover, an exploration into the effects of varying cluster numbers and depths revealed that six cluster levels yielded optimal results on the C2 dataset. This research significantly contributes to mitigating the IoT botnet threat, offering a scalable and effective solution for diverse IoT ecosystems.

Inference of Gene Regulatory Networks Based on Multi-view Hierarchical Hypergraphs.

Since gene regulation is a complex process in which multiple genes act simultaneously, accurately inferring gene regulatory networks (GRNs) is a long-standing challenge in systems biology. Although graph neural networks can formally describe intricate gene expression mechanisms, current GRN inference methods based on graph learning regard only transcription factor (TF)-target gene interactions as pairwise relationships, and cannot model the many-to-many high-order regulatory patterns that prevail among genes. Moreover, these methods often rely on limited prior regulatory knowledge, ignoring the structural information of GRNs in gene expression profiles. Therefore, we propose a multi-view hierarchical hypergraphs GRN (MHHGRN) inference model. Specifically, multiple heterogeneous biological information is integrated to construct multi-view hierarchical hypergraphs of TFs and target genes, using hypergraph convolution networks to model higher order complex regulatory relationships. Meanwhile, the coupled information diffusion mechanism and the cross-domain messaging mechanism facilitate the information sharing between genes to optimise gene embedding representations. Finally, a unique channel attention mechanism is used to adaptively learn feature representations from multiple views for GRN inference. Experimental results show that MHHGRN achieves better results than the baseline methods on the E. coli and S. cerevisiae benchmark datasets of the DREAM5 challenge, and it has excellent cross-species generalization, achieving comparable or better performance on scRNA-seq datasets from five mouse and two human cell lines.

Leveraging Dual Gloss Encoders in Chinese Biomedical Entity Linking

Entity linking is the task of assigning a unique identity to named entities mentioned in a text, a sort of word sense disambiguation that focuses on automatically determining a pre-defined sense for a target entity to be disambiguated. This study proposes the DGE (Dual Gloss Encoders) model for Chinese entity linking in the biomedical domain. We separately model a dual encoder architecture, comprising a context-aware gloss encoder and a lexical gloss encoder, for contextualized embedding representations. DGE are then jointly optimized to assign the nearest gloss with the highest score for target entity disambiguation. The experimental datasets consist of a total of 10,218 sentences that were manually annotated with glosses defined in the BabelNet 5.0 across 40 distinct biomedical entities. Experimental results show that the DGE model achieved an F1-score of 97.81, outperforming other existing methods. A series of model analyses indicate that the proposed approach is effective for Chinese biomedical entity linking.

SR-MACL: Session-Based Recommendation with Multi-layer Aggregation Augmentation in Contrastive Learning

Session-based recommendation (SBR) aims to predict the next item of interest in chronological order based on a given sequence of short-term behaviour of anonymous users. Due to the limited data available for short-term user interactions, its performance is more susceptible to data sparsity problems than traditional recommendation methods. Contrastive learning is often used to solve the data sparsity problem due to its ability to extract general features from the raw data. Existing session-based recommendation methods based on graph contrastive learning typically build graph contrastive learning by using information from other sessions to generate augmented views. While this avoids the problem that the use of dropout in traditional contrast learning methods can cause damage to the session context, it inevitably introduces irrelevant item information, which interferes with accurately modelling user interests and leads to sub-optimal model performance. To address these issues, we propose a new session recommendation method based on multi-layer aggregation augmentation contrastive learning, namely SR-MACL. In SR-MACL we construct a contrastive view by adding noise to the embedding representation and forming a contrastive embedding representation by multi-layer aggregation, which not only effectively solves the problem that traditional graph enhancement methods can destroy the context of the whole session, but also avoids the interference of irrelevant items. Experimental results on three real datasets have shown that SR-MACL can improve the accuracy of recommendation results and predict the user's next interaction more effectively.

A Dual-Attention Learning Network With Word and Sentence Embedding for Medical Visual Question Answering.

Research in medical visual question answering (MVQA) can contribute to the development of computer-aided diagnosis. MVQA is a task that aims to predict accurate and convincing answers based on given medical images and associated natural language questions. This task requires extracting medical knowledge-rich feature content and making fine-grained understandings of them. Therefore, constructing an effective feature extraction and understanding scheme are keys to modeling. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text, such as medical concepts and domain-specific terms. Meanwhile, some visual and textual feature understanding schemes cannot effectively capture the correlation between regions and keywords for reasonable visual reasoning. In this study, a dual-attention learning network with word and sentence embedding (DALNet-WSE) is proposed. We design a module, transformer with sentence embedding (TSE), to extract a double embedding representation of questions containing keywords and medical information. A dual-attention learning (DAL) module consisting of self-attention and guided attention is proposed to model intensive intramodal and intermodal interactions. With multiple DAL modules (DALs), learning visual and textual co-attention can increase the granularity of understanding and improve visual reasoning. Experimental results on the ImageCLEF 2019 VQA-MED (VQA-MED 2019) and VQA-RAD datasets demonstrate that our proposed method outperforms previous state-of-the-art methods. According to the ablation studies and Grad-CAM maps, DALNet-WSE can extract rich textual information and has strong visual reasoning ability.

O2 supplementation disambiguation in clinical narratives to support retrospective COVID-19 studies

BackgroundOxygen saturation, a key indicator of COVID-19 severity, poses challenges, especially in cases of silent hypoxemia. Electronic health records (EHRs) often contain supplemental oxygen information within clinical narratives. Streamlining patient identification based on oxygen levels is crucial for COVID-19 research, underscoring the need for automated classifiers in discharge summaries to ease the manual review burden on physicians.MethodWe analysed text lines extracted from anonymised COVID-19 patient discharge summaries in German to perform a binary classification task, differentiating patients who received oxygen supplementation and those who did not. Various machine learning (ML) algorithms, including classical ML to deep learning (DL) models, were compared. Classifier decisions were explained using Local Interpretable Model-agnostic Explanations (LIME), which visualize the model decisions.ResultClassical ML to DL models achieved comparable performance in classification, with an F-measure varying between 0.942 and 0.955, whereas the classical ML approaches were faster. Visualisation of embedding representation of input data reveals notable variations in the encoding patterns between classic and DL encoders. Furthermore, LIME explanations provide insights into the most relevant features at token level that contribute to these observed differences.ConclusionDespite a general tendency towards deep learning, these use cases show that classical approaches yield comparable results at lower computational cost. Model prediction explanations using LIME in textual and visual layouts provided a qualitative explanation for the model performance.

Graph Representation Learning-Based Fixed-Length Clinical Feature Vector Generation from Heterogeneous Medical Records.

Transformation of patient data extracted from a database into fixed-length numerical vectors requires expertise in topical medical knowledge as well as data manipulation-thus, manual feature design is labor-intensive. In this study, we propose a machine learning-based method to for this purpose applicable to electronic medical data recorded during hospitalization, which utilizes unsupervised feature extraction based on graph embedding. Unsupervised learning is performed on a heterogeneous graph using Graph2Vec, and the inclusion of clinically useful data in the obtained embedding representation is evaluated by predicting readmission within 30 days of discharge based on it. The embedded representations are observed to improve predictive performance significantly as the information contained in the graph increases, indicating the suitability of the proposed method for feature design corresponding to clinical information.

Assessing the Impact of Static, Contextual and Character Embeddings for Arabic Machine Translation

Word embeddings/representations are an important component of Natural Language Processing (NLP) tasks. Most Neural Machine Translation (NMT) systems that use such representations disregard word morphology by assigning a unique vector to each unique word in the used vocabulary and thus cannot handle Out-Of-Vocabulary (OOV) words. In some languages, such as Arabic, the meaning of words is associated with the meaning of the individual characters that constitute them, as these characters embody internal information. In this study, a combination of character- and word- level models is used to determine the most effective approaches to semantically and morphologically representing affective Arabic words. Furthermore, this work examines the strategy of combining static, character and contextual word embeddings to obtain richer representations for the Arabic Machine Translation (MT) task. To the best of our knowledge, we are the first to investigate the combination of static word embeddings, contextual ones and character-level representation in Arabic MT. Furthermore, a Deep Learning (DL) architecture is employed on data preprocessed by various prominent preprocessing techniques. Various experiments were conducted and the findings indicate that the integration of various models for word embedding and character-level representation is feasible and more effective than the state-of-the-art Arabic MT systems.

A non-aligned translation with a neoplastic classifier regularization to include vascular NBI patterns in standard colonoscopies

Polyp vascular patterns are key to categorizing colorectal cancer malignancy. These patterns are typically observed in situ from specialized narrow-band images (NBI). Nonetheless, such vascular characterization is lost from standard colonoscopies (the primary attention mechanism). Besides, even for NBI observations, the categorization remains biased for expert observations, reporting errors in classification from 59.5% to 84.2%. This work introduces an end-to-end computational strategy to enhance in situ standard colonoscopy observations, including vascular patterns typically observed from NBI mechanisms. These retrieved synthetic images are achieved by adjusting a deep representation under a non-aligned translation task from optical colonoscopy (OC) to NBI. The introduced scheme includes an architecture to discriminate enhanced neoplastic patterns achieving a remarkable separation into the embedding representation. The proposed approach was validated in a public dataset with a total of 76 sequences, including standard optical sequences and the respective NBI observations. The enhanced optical sequences were automatically classified among adenomas and hyperplastic samples achieving an F1-score of 0.86%. To measure the sensibility capability of the proposed approach, serrated samples were projected to the trained architecture. In this experiment, statistical differences from three classes with a ρ-value <0.05 were reported, following a Mann–Whitney U test. This work showed remarkable polyp discrimination results in enhancing OC sequences regarding typical NBI patterns. This method also learns polyp class distributions under the unpaired criteria (close to real practice), with the capability to separate serrated samples from adenomas and hyperplastic ones.

ScMMT: a multi-use deep learning approach for cell annotation, protein prediction and embedding in single-cell RNA-seq data.

Accurate cell type annotation in single-cell RNA-sequencing data is essential for advancing biological and medical research, particularly in understanding disease progression and tumor microenvironments. However, existing methods are constrained by single feature extraction approaches, lack of adaptability to immune cell types with similar molecular profiles but distinct functions and a failure to account for the impact of cell label noise on model accuracy, all of which compromise the precision of annotation. To address these challenges, we developed a supervised approach called scMMT. We proposed a novel feature extraction technique to uncover more valuable information. Additionally, we constructed a multi-task learning framework based on the GradNorm method to enhance the recognition of challenging immune cells and reduce the impact of label noise by facilitating mutual reinforcement between cell type annotation and protein prediction tasks. Furthermore, we introduced logarithmic weighting and label smoothing mechanisms to enhance the recognition ability of rare cell types and prevent model overconfidence. Through comprehensive evaluations on multiple public datasets, scMMT has demonstrated state-of-the-art performance in various aspects including cell type annotation, rare cell identification, dropout and label noise resistance, protein expression prediction and low-dimensional embedding representation.

It’s Not Always about Wide and Deep Models: Click-Through Rate Prediction with a Customer Behavior-Embedding Representation

Alongside natural language processing and computer vision, large learning models have found their way into e-commerce. Especially, for recommender systems and click-through rate prediction, these models have shown great predictive power. In this work, we aim to predict the probability that a customer will click on a given recommendation, given only its current session. Therefore, we propose a two-stage approach consisting of a customer behavior-embedding representation and a recurrent neural network. In the first stage, we train a self-supervised skip-gram embedding on customer activity data. The resulting embedding representation is used in the second stage to encode the customer sequences which are then used as input to the learning model. Our proposed approach diverges from the prevailing trend of utilizing extensive end-to-end models for click-through rate prediction. The experiments, which incorporate a real-world industrial use case and a widely used as well as openly available benchmark dataset, demonstrate that our approach outperforms the current state-of-the-art models. Our approach predicts customers’ click intention with an average F1 accuracy of 94% for the industrial use case which is one percentage point higher than the state-of-the-art baseline and an average F1 accuracy of 79% for the benchmark dataset, which outperforms the best tested state-of-the-art baseline by more than seven percentage points. The results show that, contrary to current trends in that field, large end-to-end models are not always needed. The analysis of our experiments suggests that the reason for the performance of our approach is the self-supervised pre-trained embedding of customer behavior that we use as the customer representation.

Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis

Human Multimodal Sentiment Analysis (MSA) is an attractive research that studies sentiment expressed from multiple heterogeneous modalities. While transformer-based methods have achieved great success, designing an effective “co-attention” model to associate text modality with nonverbal modalities remains challenging. There are two main problems: 1) the dominant role of the text in modalities is underutilization, and 2) the interaction between modalities is not sufficiently explored. This paper proposes a deep modular Co-Attention Shifting Network (CoASN) for MSA. A Cross-modal Modulation Module based on Co-attention (CMMC) and an Advanced Modality-mixing Adaptation Gate (AMAG) are constructed. The CMMC consists of the Text-guided Co-Attention (TCA) and Interior Transformer Encoder (ITE) units to capture inter-modal features and intra-modal features. With text modality as the core, the CMMC module aims to guide and promote the expression of emotion in nonverbal modalities, and the nonverbal modalities increase the richness of the text-based multimodal sentiment information. In addition, the AMAG module is introduced to explore the dynamical correlations among all modalities. Particularly, this efficient module first captures the nonverbal shifted representations and then combines them to calculate the shifted word embedding representations for the final MSA tasks. Extensive experiments on two commonly used datasets, CMU-MOSI and CMU-MOSEI, demonstrate that our proposed method is superior to the state-of-the-art performance.

VERB: Visualizing and Interpreting Bias Mitigation Techniques Geometrically for Word Representations

Word vector embeddings have been shown to contain and amplify biases in the data they are extracted from. Consequently, many techniques have been proposed to identify, mitigate, and attenuate these biases in word representations. In this article, we utilize interactive visualization to increase the interpretability and accessibility of a collection of state-of-the-art debiasing techniques. To aid this, we present the Visualization of Embedding Representations for deBiasing (VERB) system, an open-source web-based visualization tool that helps users gain a technical understanding and visual intuition of the inner workings of debiasing techniques, with a focus on their geometric properties. In particular, VERB offers easy-to-follow examples that explore the effects of these debiasing techniques on the geometry of high-dimensional word vectors. To help understand how various debiasing techniques change the underlying geometry, VERB decomposes each technique into interpretable sequences of primitive transformations and highlights their effect on the word vectors using dimensionality reduction and interactive visual exploration. VERB is designed to target natural language processing (NLP) practitioners who are designing decision-making systems on top of word embeddings and researchers working with the fairness and ethics of machine learning systems in NLP. It can also serve as a visual medium for education, which helps an NLP novice understand and mitigate biases in word embeddings.

Deep graph tensor learning for temporal link prediction

Knowledge discovery on dynamic graphs has received much attention in recent years. As a key task of dynamic graph research, the goal of temporal link prediction is to accurately predict the time-varying links in dynamic networks. Uncertainty in link emergence is a major challenge in this research, as it is not easy to learn stable and reliable link-level feature representations, which are usually readily available on static graphs. In order to adapt to the ever-changing graph structure, this paper proposes to construct a deep graph tensor learning model, which can capture the contextual characteristics of graph evolution from both the graph structure (spatial) mode and the link sequence (temporal) mode. Therefore, compared to link prediction on static graphs, temporal link prediction can benefit more from the link-level embedding representations coupled with spatio-temporal features. The experimental results on seven public dynamic graph datasets show that the prediction accuracy obtained by the new model is overall better than competing models such as GC-LSTM, EvolveGCN, and HTGN. In the meantime, as a result of getting rid of the traditional RNN learning paradigm, the new model is also significantly better than the traditional temporal graph learning model in terms of training efficiency.

Contrastive Graph Similarity Networks

Graph similarity learning is a significant and fundamental issue in the theory and analysis of graphs, which has been applied in a variety of fields, including object tracking, recommender systems, similarity search, and so on. Recent methods for graph similarity learning that utilize deep learning typically share two deficiencies: (1) they leverage graph neural networks as backbones for learning graph representations but have not well captured the complex information inside data, and (2) they employ a cross-graph attention mechanism for graph similarity learning, which is computationally expensive. Taking these limitations into consideration, a method for graph similarity learning is devised in this study, namely, Contrastive Graph Similarity Network (CGSim). To enhance graph similarity learning, CGSim makes use of the complementary information of two input graphs and captures pairwise relations in a contrastive learning framework. By developing a dual contrastive learning module with a node-graph matching and a graph-graph matching mechanism, our method significantly reduces the quadratic time complexity for cross-graph interaction modeling to linear time complexity. Jointly learning in an end-to-end framework, the graph representation embedding module and the well-designed contrastive learning module can be beneficial to one another. A comprehensive series of experiments indicate that CGSim outperforms state-of-the-art baselines on six datasets and significantly reduces the computational cost, which demonstrates our CGSim model’s superiority over other baselines.

AntiViralDL: Computational Antiviral Drug Repurposing Using Graph Neural Network and Self-Supervised Learning.

Viral infections have emerged as significant public health concerns for decades. Antiviral drugs, specifically designed to combat these infections, have the potential to reduce the disease burden substantially. However, traditional drug development methods, based on biological experiments, are resource-intensive, time-consuming, and low efficiency. Therefore, computational approaches for identifying antiviral drugs can enhance drug development efficiency. In this study, we introduce AntiViralDL, a computational framework for predicting virus-drug associations using self-supervised learning. Initially, we construct a reliable virus-drug association dataset by integrating the existing Drugvirus2 database and FDA-approved virus-drug associations. Utilizing these two datasets, we create a virus-drug association bipartite graph and employ the Light Graph Convolutional Network (LightGCN) to learn embedding representations of viruses and drugs. To address the sparsity of virus-drug association pairs, AntiViralDL incorporates contrastive learning to improve prediction accuracy. We implement data augmentation by adding random noise to the embedding representation space of virus and drug nodes, as opposed to traditional edge and node dropout. Finally, we calculate an inner product to predict virus-drug association relationships. Experimental results reveal that AntiViralDL achieves AUC and AUPR values of 0.8450 and 0.8494, respectively, outperforming four benchmarked virus-drug association prediction models. The case study further highlights the efficacy of AntiViralDL in predicting anti-COVID-19 drug candidates.

HuBar: A Visual Analytics Tool to Explore Human Behavior based on fNIRS in AR Guidance Systems.

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.