Embedding Model Research Articles

Attention-aware with stacked embedding for sentiment analysis of student feedback through deep learning techniques.

Automatic polarity prediction is a challenging assessment issue. Even though polarity assessment is a critical topic with many existing applications, it is probably not an easy challenge and faces several difficulties in natural language processing (NLP). Public polling data can give useful information, and polarity assessment or classification of comments on Twitter and Facebook may be an effective approach for gaining a better understanding of user sentiments. Text embedding techniques and models related to the artificial intelligence field and sub-fields with differing and almost accurate parameters are among the approaches available for assessing student comments. Existing state-of-the-art methodologies for sentiment analysis to analyze student responses were discussed in this study endeavor. An innovative hybrid model is proposed that uses ensemble learning-based text embedding, a multi-head attention mechanism, and a combination of deep learning classifiers. The proposed model outperforms the existing state-of-the-art deep learning-based techniques. The proposed model achieves 95% accuracy, 97% recall, having a precision of 95% with an F1-score of 96% demonstrating its effectiveness in sentiment analysis of student feedback.

Accurate and efficient protein embedding using multi-teacher distillation learning.

Protein embedding, which represents proteins as numerical vectors, is a crucial step in various learning-based protein annotation/classification problems, including gene ontology prediction, protein-protein interaction prediction, and protein structure prediction. However, existing protein embedding methods are often computationally expensive due to their large number of parameters, which can reach millions or even billions. The growing availability of large-scale protein datasets and the need for efficient analysis tools have created a pressing demand for efficient protein embedding methods. We propose a novel protein embedding approach based on multi-teacher distillation learning, which leverages the knowledge of multiple pre-trained protein embedding models to learn a compact and informative representation of proteins. Our method achieves comparable performance to state-of-the-art methods while significantly reducing computational costs and resource requirements. Specifically, our approach reduces computational time by ∼70% and maintains ±1.5% accuracy as the original large models. This makes our method well-suited for large-scale protein analysis and enables the bioinformatics community to perform protein embedding tasks more efficiently. The source code of MTDP is available via https://github.com/KennthShang/MTDP.

Fast clustering and cell-type annotation of scATAC data using pre-trained embeddings.

Data from the single-cell assay for transposase-accessible chromatin using sequencing (scATAC-seq) are now widely available. One major computational challenge is dealing with high dimensionality and inherent sparsity, which is typically addressed by producing lower dimensional representations of single cells for downstream clustering tasks. Current approaches produce such individual cell embeddings directly through a one-step learning process. Here, we propose an alternative approach by building embedding models pre-trained on reference data. We argue that this provides a more flexible analysis workflow that also has computational performance advantages through transfer learning. We implemented our approach in scEmbed, an unsupervised machine-learning framework that learns low-dimensional embeddings of genomic regulatory regions to represent and analyze scATAC-seq data. scEmbed performs well in terms of clustering ability and has the key advantage of learning patterns of region co-occurrence that can be transferred to other, unseen datasets. Moreover, models pre-trained on reference data can be exploited to build fast and accurate cell-type annotation systems without the need for other data modalities. scEmbed is implemented in Python and it is available to download from GitHub. We also make our pre-trained models available on huggingface for public use. scEmbed is open source and available at https://github.com/databio/geniml. Pre-trained models from this work can be obtained on huggingface: https://huggingface.co/databio.

Accelerating scientific discovery with generative knowledge extraction, graph-based representation, and multimodal intelligent graph reasoning

Abstract Leveraging generative Artificial Intelligence (AI), we have transformed a dataset comprising 1,000 scientific papers focused on biological materials into a comprehensive ontological knowledge graph. Through an in-depth structural analysis of this graph, we have calculated node degrees, identified communities along with their connectivities, and evaluated clustering coefficients and betweenness centrality of pivotal nodes, uncovering fascinating knowledge architectures. We find that the graph has an inherently scale-free nature, shows a high level of connectedness, and can be used as a rich source for downstream graph reasoning by taking advantage of transitive and isomorphic properties to reveal insights into unprecedented interdisciplinary relationships that can be used to answer queries, identify gaps in knowledge, propose never-before-seen material designs, and predict material behaviors. Using a large language embedding model we compute deep node representations and use combinatorial node similarity ranking to develop a path sampling strategy that allows us to link dissimilar concepts that have previously not been related. One comparison revealed detailed structural parallels between biological materials and Beethoven's 9th Symphony, highlighting shared patterns of complexity through isomorphic mapping. In another example, the algorithm proposed an innovative hierarchical mycelium-based composite based on integrating path sampling with principles extracted from Kandinsky's `Composition VII' painting. The resulting material integrates an innovative set of concepts that include a balance of chaos and order, adjustable porosity, mechanical strength, and complex patterned chemical functionalization. We uncover other isomorphisms across science, technology and art, revealing a nuanced ontology of immanence that reveal a context-dependent heterarchical interplay of constituents. Because our method transcends established disciplinary boundaries through diverse data modalities (graphs, images, text, numerical data, etc.), graph-based generative AI achieves a far higher degree of novelty, explorative capacity, and technical detail, than conventional approaches and establishes a widely useful framework for innovation by revealing hidden connections.

Learning Joint Topic Representation for Detecting Drift in Social Media Text

Social media texts like tweets and blogs are collaboratively created by human interaction. Rapidly changing trends are leading to topic drift in the social media text. This drift is usually associated with words and hashtags. However, geotags play an essential part in determining topic distribution with location context. The rate of change in the distribution of words, hashtags and geotags cannot be considered uniform and must be handled accordingly. This paper builds a topic model that associates the topic with a mixture of distributions of words, hashtags and geotags. Stochastic gradient Langevin dynamic model with varying mini-batch sizes is used to capture the changes due to the asynchronous distribution of words and tags. Topic representations with co-occurrence and location contexts are specified as hashtag context vector and geotag context vector respectively. These two vectors are jointly learned to yield topical word embedding vectors over time conditioned on hashtags and geotags that can predict location-based topical variations effectively. When evaluated with Chennai and UK geolocated Twitter data, the proposed joint topical word embedding model enhanced by the social tags context, outperforms other methods.

VNEavXT: Offline Virtual Network Embedding Model Considering Crosstalk-Avoided Approach in Spectrally-Spatially Elastic Optical Networks

VNEavXT: Offline Virtual Network Embedding Model Considering Crosstalk-Avoided Approach in Spectrally-Spatially Elastic Optical Networks

Interactive Feature Embedding for Infrared and Visible Image Fusion.

General deep learning-based methods for infrared and visible image fusion rely on the unsupervised mechanism for vital information retention by utilizing elaborately designed loss functions. However, the unsupervised mechanism depends on a well-designed loss function, which cannot guarantee that all vital information of source images is sufficiently extracted. In this work, we propose a novel interactive feature embedding in a self-supervised learning framework for infrared and visible image fusion, attempting to overcome the issue of vital information degradation. With the help of a self-supervised learning framework, hierarchical representations of source images can be efficiently extracted. In particular, interactive feature embedding models are tactfully designed to build a bridge between self-supervised learning and infrared and visible image fusion learning, achieving vital information retention. Qualitative and quantitative evaluations exhibit that the proposed method performs favorably against state-of-the-art methods.

EHPR: Learning evolutionary hierarchy perception representation based on quaternion for temporal knowledge graph completion

Research on temporal knowledge graphs garners attention due to the intricate connection between facts and dynamic temporal factors. However, existing research uses timestamp as auxiliary data for representation learning and directly integrate it into facts, resulting in the inability to capture the intrinsic connections between relations under time evolution. To handle these challenges, we propose the Evolutionary Hierarchy Perception Representation (EHPR), which first leverages the Hamilton product to perform rotational transformations on relation and entity over time, aiming to learn temporal relation and temporal entity with close interactions with time information. Later, EHPR is divided into two modules: (a) Rotating the head entity towards the tail entity using temporal relation through Hamilton product to model complex patterns with quaternion rotation capabilities. (b) Adopting an evolutionary hierarchical factor to capture the differences in modulus distribution between the temporal head entity and the temporal tail entity, aiming to manage the evolutionary hierarchical information between different temporal entities. In this way, EHPR not only utilizes the rich quaternion rotation capabilities to model various relation patterns but also further enables modeling of evolutionary hierarchical patterns through evolutionary hierarchy factors. Experiments show that EHPR achieves remarkable performance on six mature benchmarks compared to state-of-the-art models. Furthermore, we successfully transferred the core idea of EHPR into complex embeddings, showcasing the framework's adaptability. Compared to complex embedding models, EHPR also demonstrates stronger expressive abilities with the Hamilton operator, surpassing the performance of complex Hermitian operator.

Multi-Scale Spatio-Temporal Attention Networks for Network-Scale Traffic Learning and Forecasting.

Accurate and timely forecasting of traffic on local road networks is crucial for deploying effective dynamic traffic control, advanced route planning, and navigation services. This task is particularly challenging due to complex spatio-temporal dependencies arising from non-Euclidean spatial relations in road networks and non-linear temporal dynamics influenced by changing road conditions. This paper introduces the spatio-temporal network embedding (STNE) model, a novel deep learning framework tailored for learning and forecasting graph-structured traffic data over extended input sequences. Unlike traditional convolutional neural networks (CNNs), the model employs graph convolutional networks (GCNs) to capture the spatial characteristics of local road network topologies. Moreover, the segmentation of very long input traffic data into multiple sub-sequences, based on significant temporal properties such as closeness, periodicity, and trend, is performed. Multi-dimensional long short-term memory neural networks (MDLSTM) are utilized to flexibly access multi-dimensional context. Experimental results demonstrate that the STNE model surpasses state-of-the-art traffic forecasting benchmarks on two large-scale real-world traffic datasets.

Integrating word embedding and topic modeling for sentiment analysis: A case study on the social mood on economy

In recent years, textual analysis and embedding spaces have become essential and complementary tools for sentiment analysis in National Statistics Institutes’ research, owing to their ability to summarize discussed topics effectively. Istat has developed an innovative tool, wordembox, which allows external users to explore the outputs of popular word embedding algorithms, such as Word2Vec and FastText. This tool enriches the analysis with a novel graph functionality, enabling users to discover clusters of words and facilitating implicit topic modeling. This article focuses on Social Mood on Economy (SME) posts over a period in which the index recorded a strong downward trend: the first month of the Russia-Ukraine conflict at the beginning of 2022. We compare findings from wordembox with standard topic modeling techniques, including Bayesian Latent Dirichlet Allocation (LDA), Top2Vec, and BERTopic, recent methods that extract clusters from word embedding spaces. These techniques show coherent results, and their combined use in textual analysis may create a synergy that enhances the informative content of synthetic indexes such as ‘Social Mood on Economy Index (SMEI)’.

TAP with ease: a generic recommendation system for trigger-action programming based on multi-modal representation learning

The escalating popularity of smart devices has given rise to an increasing trend wherein users leverage customized trigger-action programming (TAP) rules within the Internet of Things (IoT) to automate various aspects of their lives. This article addresses the challenge of effectively combining functions provided by many smart devices and online services by introducing a novel multi-modal representation learning model called TAP-TAG. This model integrates both textual and graph structures inherent in TAP rules, offering a holistic method to rule recommendation. TAP-TAG comprises two branches: the Knowledge Graph Embedding model, which projects triplets extracted from the TAP dataset into embeddings, and convolution neural networks that extract semantic features from the textual content of TAP rules. Extensive experiments are conducted on real-world TAP datasets to evaluate our model’s ability to recommend relevant rules based on user preferences. The experimental results show that TAP-TAG can outperform the state-of-the-art method by 5% in Precision@5, indicating that TAP-TAG is highly effective in providing accurate and diverse recommendations for TAP rules.

Moral attitudes towards effort and efficiency: a comparison between American and Chinese history

In some cultures, merely exerting effort is considered virtuous, even when the effort is inefficient. Our study examines how this moral attitude towards effort (relative to efficiency) has evolved historically across two distinct sociopolitical and linguistic contexts: the People’s Republic of China and the United States, using natural language processing techniques. Specifically, two formal political corpora were used—the People’s Daily (1950–2021) and the Congressional speeches for the U.S. (1873–2011). We developed dictionaries for each concept based on pre-trained word embedding models in both languages. Moral attitudes towards effort and efficiency were calculated on a year-by-year basis as the cosine similarity between the dictionaries of these concepts and an existing dictionary of morality. We benchmarked the fluctuations of moral attitude towards inefficient effort against critical historical events in the two countries. Additional time series analysis and Granger tests revealed the association and potential directionality between the evolution of moral attitude towards inefficient effort and critical socio-cultural variables such as collectivism and cultural looseness. Our research sheds light on the historical and socio-cultural roots of moralization of effort and has implications for historical psychology research on moral attitudes.

REDAffectiveLM: leveraging affect enriched embedding and transformer-based neural language model for readers’ emotion detection

REDAffectiveLM: leveraging affect enriched embedding and transformer-based neural language model for readers’ emotion detection

Scupa: Single-cell unified polarization assessment of immune cells using the single-cell foundation model.

Immune cells undergo cytokine-driven polarization in respond to diverse stimuli. This process significantly modulates their transcriptional profiles and functional states. Although single-cell RNA sequencing (scRNA-seq) has advanced our understanding of immune responses across various diseases or conditions, currently there lacks a method to systematically examine cytokine effects and immune cell polarization. To address this gap, we developed Single-cell unified polarization assessment (Scupa), the first computational method for comprehensive immune cell polarization analysis. Scupa is trained on data from the Immune Dictionary, which characterizes 66 cytokine-driven polarization states across 14 immune cell types. By leveraging the cell embeddings from the Universal Cell Embeddings model, Scupa effectively identifies polarized cells in new datasets generated from different species and experimental conditions. Applications of Scupa in independent datasets demonstrated its accuracy in classifying polarized cells and further revealed distinct polarization profiles in tumor-infiltrating myeloid cells across cancers. Scupa complements conventional single-cell data analysis by providing new insights into immune cell polarization, and it holds promise for assessing molecular effects or identifying therapeutic targets in cytokine-based therapies.

Self-X heterogeneous attributed graph embedding-based product configuration framework for cognitive mass personalization

Cognitive mass personalization (CMP) is a promising manufacturing paradigm; equipped with cognitive capabilities like reasoning, CMP satisfies changeable needs via configuring personalized products at scale. In CMP, knowledge graphs (KGs) are exploited by smart product-service systems (SPSS) to support cognitive configuration/reconfiguration processes. However, the extant KG-enabled SPSSs are built upon fixed configurations and hybrid frameworks due to lacking a graph embedding (GE) model to render cognitive configuration decisions. In fact, GE is scarcely used in SPSS configuration, because it is not only compromised by the heterogeneity of KGs entailed by content-related specifications and complex structures but also influenced by the feature randomness and feature drift problems, which are triggered by accumulative errors and inconsistent objectives due to noisy assignments and different configuration tasks, separately. To address these limitations, a Self-X Heterogeneous Attributed Graph Embedding (SXHAGE) model is proposed in a Self-X architecture, which includes 1) self-attention graph attention networks, 2) a self-adaptive autoencoder, and 3) self-optimizing training objectives, to present heterogeneous data through jointly optimizing heterogeneous attributed entities and relations. A systematic SXHAGE-based configuration framework, in which product family design and configuration recommending are enabled by graph clustering and link prediction, is developed as a continuous updating loop to proactively configure personalized products. A real-world case study, i.e., configure personalized electric clippers via a web-based sustainable configuration platform, is performed to validate the applicability of the proposed framework in the CMP context. Moreover, extensive experiments on the case study dataset demonstrate the superiority of SXHAGE over the state-of-the-art algorithms, e.g., surpassing Deep Neighbor-Aware Embedding (DNENC) by 18 % in F1-score for graph clustering and by 5 % in ROC-AUC for link prediction.

Cumulative rib fracture risk after stereotactic body radiotherapy in patients with localized non-small cell lung cancer

IntroductionRib fracture is a known complication after stereotactic body radiotherapy (SBRT). Patient-related parameters are essential to provide patient-tailored risk estimation, however, their impact on rib fracture is less documented compared to dosimetric parameters. This study aimed to predict the risk of rib fractures in patients with localized non-small cell lung cancer (NSCLC) post-SBRT based on both patient-related and dosimetric parameters with death as a competing risk. Materials and methodsIn total, 602 patients with localized NSCLC treated with SBRT between 2010–2020 at Odense University Hospital, Denmark were included. All patients received SBRT with 45–66 Gray (Gy)/3 fractions. Rib fractures were identified in CT-scans using a word embedding model. The cumulative incidence function was based on cause-specific Cox hazard models with variable selection based on cross-validation model likelihood performed using 50 bootstraps. ResultsIn total, 19 % of patients experienced a rib fracture. The cumulative risk of rib fracture increased rapidly from 6-54 months post-SBRT. Female gender, bone density, near max dose to the rib, V30 and V40 to the rib, gross tumor volume, and mean lung dose were significantly associated with rib fracture risk in univariable analysis. The final multi-variable model consisted of V20 and V30 to the rib and mean lung dose. ConclusionFemale gender and low bone density in male patients are significant predictors of rib fracture risk. The final model predicting cumulative rib fracture risk of 19 % in patients with localized NSCLC treated with SBRT contained no patient-related parameters, suggesting that dosimetric parameters are the primary drivers.

ERT-GFAN: A multimodal drug–target interaction prediction model based on molecular biology and knowledge-enhanced attention mechanism

In drug discovery, precisely identifying drug–target interactions is crucial for finding new drugs and understanding drug mechanisms. Evolving drug/target heterogeneous data presents challenges in obtaining multimodal representation in drug–target prediction(DTI). To deal with this, we propose ‘ERT-GFAN’, a multimodal drug–target interaction prediction model inspired by molecular biology. Firstly, it integrates bio-inspired principles to obtain structure feature of drugs and targets using Extended Connectivity Fingerprints(ECFP). Simultaneously, the knowledge graph embedding model RotatE is employed to discover the interaction feature of drug–target pairs. Subsequently, Transformer is utilized to refine the contextual neighborhood features from the obtained structure feature and interaction features, and multi-modal high-dimensional fusion features of the three-modal information constructed. Finally, the final DTI prediction results are outputted by integrating the multimodal fusion features into a graphical high-dimensional fusion feature attention network (GFAN) using our innovative multimodal high-dimensional fusion feature attention. This multimodal approach offers a comprehensive understanding of drug–target interactions, addressing challenges in complex knowledge graphs. By combining structure feature, interaction feature, and contextual neighborhood features, ‘ERT-GFAN’ excels in predicting DTI. Empirical evaluations on three datasets demonstrate our method’s superior performance, with AUC of 0.9739, 0.9862, and 0.9667, AUPR of 0.9598, 0.9789, and 0.9750, and Mean Reciprocal Rank(MRR) of 0.7386, 0.7035, and 0.7133. Ablation studies show over a 5% improvement in predictive performance compared to baseline unimodal and bimodal models. These results, along with detailed case studies, highlight the efficacy and robustness of our approach.

An radicals construction technique based on dual quaternions and hierarchical transformers

At present, the exploration of Knowledge Graph Embedding (KGE) mainly focuses on static KGE models. Due to the diversity and complexity of vertical domain data, it is difficult for existing KGE models to achieve excellent performance. Specific KGE models must be constructed for research on vertical domain knowledge graphs. This paper explores the relationships between the composition of Chinese characters and their sounds, meanings, and forms by constructing the Chinese Radical Knowledge Graph (CRKG). CRKG was constructed from a unique perspective according to the characteristics of Chinese radicals and knowledge graphs. The minor components of Chinese characters are separated and constructed into the fourth dimension of the knowledge graph. In this paper, we propose a new Chinese Radical Knowledge Graph Embedding (CRKGE) model, RotAL, which introduces dual quaternions and hierarchical transformers into KGE. RotAL performs a translation-rotation operation on the head entity with the relation as a parameter, then does a translation operation on the head entity with the minor component as a parameter to be close to the tail entity. Finally, we feed the trained quaternions into hierarchical transformers to jointly learn entity-related composition and relational contextualization based on a head entity’s neighborhood. We demonstrate that our method can model and infer various relational patterns of CRKG, such as symmetric/antisymmetric, reflexive, inversion, and multiple. The experimental results on four CRKG datasets demonstrate that the RotAL model achieves the highest Hit@1 score, outperforming existing KGE models in accurately identifying the target entity on the first attempt. Furthermore, the RotAL model consistently achieves superior performance across other evaluation metrics, substantiating its significant advantage over existing models.

Concept graph embedding models for enhanced accuracy and interpretability

Concept graph embedding models for enhanced accuracy and interpretability

Link prediction for hypothesis generation: an active curriculum learning infused temporal graph-based approach

Over the last few years Literature-based Discovery (LBD) has regained popularity as a means to enhance the scientific research process. The resurgent interest has spurred the development of supervised and semi-supervised machine learning models aimed at making previously implicit connections between scientific concepts/entities within often extensive bodies of literature explicit—i.e., suggesting novel scientific hypotheses. In doing so, understanding the temporally evolving interactions between these entities can provide valuable information for predicting the future development of entity relationships. However, existing methods often underutilize the latent information embedded in the temporal aspects of the interaction data. Motivated by applications in the food domain—where we aim to connect nutritional information with health-related benefits—we address the hypothesis-generation problem using a temporal graph-based approach. Given that hypothesis generation involves predicting future (i.e., still to be discovered) entity connections, in our view the ability to capture the dynamic evolution of connections over time is pivotal for a robust model. To address this, we introduce THiGER, a novel batch contrastive temporal node-pair embedding method. THiGER excels in providing a more expressive node-pair encoding by effectively harnessing node-pair relationships. Furthermore, we present THiGER-A, an incremental training approach that incorporates an active curriculum learning strategy to mitigate label bias arising from unobserved connections. By progressively training on increasingly challenging and high-utility samples, our approach significantly enhances the performance of the embedding model. Empirical validation of our proposed method demonstrates its effectiveness on established temporal-graph benchmark datasets, as well as on real-world datasets within the food domain.