Low-dimensional Dense Vectors Research Articles

A knowledge graph embedding model based attention mechanism for enhanced node information integration.

The purpose of knowledge embedding is to extract entities and relations from the knowledge graph into low-dimensional dense vectors, in order to be applied to downstream tasks, such as connection prediction and intelligent classification. Existing knowledge embedding methods still have many limitations, such as the contradiction between the vast amount of data and limited computing power, and the challenge of effectively representing rare entities. This article proposed a knowledge embedding learning model, which incorporates a graph attention mechanism to integrate key node information. It can effectively aggregate key information from the global graph structure, shield redundant information, and represent rare nodes in the knowledge base independently of its own structure. We introduce a relation update layer to further update the relation based on the results of entity training. The experiment shows that our method matches or surpasses the performance of other baseline models in link prediction on the FB15K-237 dataset. The metric Hits@1 has increased by 10.9% compared to the second-ranked baseline model. In addition, we conducted further analysis on rare nodes with fewer neighborhoods, confirming that our model can embed rare nodes more accurately than the baseline models.

Partial order relation-based gene ontology embedding improves protein function prediction.

Protein annotation has long been a challenging task in computational biology. Gene Ontology (GO) has become one of the most popular frameworks to describe protein functions and their relationships. Prediction of a protein annotation with proper GO terms demands high-quality GO term representation learning, which aims to learn a low-dimensional dense vector representation with accompanying semantic meaning for each functional label, also known as embedding. However, existing GO term embedding methods, which mainly take into account ancestral co-occurrence information, have yet to capture the full topological information in the GO-directed acyclic graph (DAG). In this study, we propose a novel GO term representation learning method, PO2Vec, to utilize the partial order relationships to improve the GO term representations. Extensive evaluations show that PO2Vec achieves better outcomes than existing embedding methods in a variety of downstream biological tasks. Based on PO2Vec, we further developed a new protein function prediction method PO2GO, which demonstrates superior performance measured in multiple metrics and annotation specificity as well as few-shot prediction capability in the benchmarks. These results suggest that the high-quality representation of GO structure is critical for diverse biological tasks including computational protein annotation.

MMCPP: A MULTI-MODAL CONTRASTIVE PRE-TRAINING MODEL FOR PLACE REPRESENTATION BASED ON THE SPATIO-TEMPORAL FRAMEWORK

Abstract. The concept of "place" is crucial for understanding geographical environments from a human perspective. Place representation learning involves converting places into numerical low-dimensional dense vectors and is a fundamental procedure for artificial intelligence in geography (GeoAI). However, most studies ignore multi-level distance constraints and spatial proximity interactions that enable behavioral interactions between places. Furthermore, representing the temporal characteristics of these interactions in trajectory sequences poses a challenge for natural language processing and other field techniques. In addition, most existing methods rely on all modalities from inputs as they use joint training to integrate multiple modalities. To address these issues, we propose a Multi-Modal Contrastive Pre-training model for Place representation (MMCPP). Our model consists of three encoders that capture corresponding place attributes across different modalities, including point of interests (POIs), images, and trajectories. The trajectory encoder, named RodtFormer, takes fine-grained spatio-temporal trajectories as input and leverages self-attention with rotary temporal interval position embedding to simulate dynamic spatial and behavioral proximity interactions between places. By using a coordinated pre-training framework, MMCPP independently encodes place representations across different modalities and improves model reusability. We verify the effectiveness of our model on a taxi trajectory dataset using the location prediction task at next n seconds, including 30 seconds(s), 180(s), 300(s). Our results demonstrate that compared to existing embedding methods, our model is capable of learning higher-quality position representations during pre-training, leading to improved performance on downstream tasks.

Transformer based contextual text representation framework for intelligent information retrieval

With the advent of transformer-based architectures, the contextual representation of text data has leveraged the query and the document to be represented in low-dimensional dense vector space. These vectors are learned embeddings of fixed sizes, resulting in deeper text understanding. In this study, we designed a pipeline for effectively retrieving documents from a large search space by combining the deeper text understanding capabilities of the transformer-based BERT model and a phrase embedding-based query expansion model. To learn the contextual representations, we fine-tuned a deep semantic matching model by separately encoding the document and the query. The encoder model is based on the Sentence BERT (SBERT) architecture, which separately generates dense vector representations of documents and queries. The study has also addressed the maximum token length limitation of transformer-based models through the summarization of lengthy documents. In addition, to improve the clarity and completeness of short queries and reduce the semantic gap, a phrase embedding-based query expansion model is employed. The documents and their dense vectors are indexed using the Elasticsearch engine, and matched them with query vectors for retrieving query-specific documents. Finally, the BERT-based cross-encoder model is used to re-rank the relevant records for each query. It performs full self-attention over the inputs, and yields richer text interactions to produce the final results. To assess performance, experiments are conducted on two well-known datasets, TREC-CDS-2014 and OHSUMED. A comparative analysis is carried out, which clearly demonstrates that the proposed framework produced competitive retrieval results.

HMSG: Heterogeneous graph neural network based on Metapath SubGraph learning

Heterogeneous graph neural network (HGNN) models, capable of learning low-dimensional dense vectors from heterogeneous graphs for downstream graph-mining tasks, have attracted increasing attention in recent years. For these models, metapath-based methods have been widely adopted. However, most existing metapath-based HGNN models either discard intermediate nodes within a metapath, resulting in information loss, or indiscriminately aggregate information along a metapath containing different types of nodes, resulting in unavoidable learning bias. To overcome these limitations, a new HGNN model named HMSG, is proposed in this paper to comprehensively capture structural, semantic and attribute information from both homogeneous and heterogeneous neighbors more purposefully. To achieve this, a type-specific linear transformation is first applied to transfer the node attributes to different types of nodes with the same latent factor space. In the new model, the heterogeneous graph is decomposed into multiple metapath-based homogeneous and heterogeneous subgraphs where each subgraph associates specific semantic and structural information; this is different from existing models, which mainly rely on symmetric metapaths. Subsequently, tailored attention-based message aggregation methods are independently applied to each subgraph such that information learning can be more targeted. Finally, information from different subgraphs is fused through graph-level attention to obtain a complete representation. The learned representations are evaluated by several graph-mining tasks. Results indicate that the HMSG attains the best performance in all evaluation metrics than state-of-the-art baselines. Further ablation experiments demonstrate the effectiveness of the modules designed for the HMSG.

Recognition and Analysis of Scene-Emotion in Photographic Works Based on AI Technology

Emotional effect is highly subjective in people's cognitive process, and a single discrete emotional feeling can hardly support the description of the immersion scene, which also puts forward higher requirements for emotional calculation in photography. Therefore, this article first constructs a photographic scene recognition model, and then establishes a visual emotion analysis model which optimizes the basic structure of vgg19 through CNN, extracts the user's photography situation information from the corresponding image metadata, establishes the mapping relationship between situation and emotion, and obtains the low-dimensional dense vector representation of the situation features through embedding. The authors divided eight emotional categories; accuracy of the model is compared and the feature distribution of scene-emotion in different works is analyzed. The results show that the accuracy of the scene-emotion recognition model of photographic works after multimodal fusion is high, reaching 73.9%, in addition, different shooting scenes can distinguish the emotional characteristics of works.

Drug repurposing and prediction of multiple interaction types via graph embedding

BackgroundFinding drugs that can interact with a specific target to induce a desired therapeutic outcome is key deliverable in drug discovery for targeted treatment. Therefore, both identifying new drug–target links, as well as delineating the type of drug interaction, are important in drug repurposing studies.ResultsA computational drug repurposing approach was proposed to predict novel drug–target interactions (DTIs), as well as to predict the type of interaction induced. The methodology is based on mining a heterogeneous graph that integrates drug–drug and protein–protein similarity networks, together with verified drug-disease and protein-disease associations. In order to extract appropriate features, the three-layer heterogeneous graph was mapped to low dimensional vectors using node embedding principles. The DTI prediction problem was formulated as a multi-label, multi-class classification task, aiming to determine drug modes of action. DTIs were defined by concatenating pairs of drug and target vectors extracted from graph embedding, which were used as input to classification via gradient boosted trees, where a model is trained to predict the type of interaction. After validating the prediction ability of DT2Vec+, a comprehensive analysis of all unknown DTIs was conducted to predict the degree and type of interaction. Finally, the model was applied to propose potential approved drugs to target cancer-specific biomarkers.ConclusionDT2Vec+ showed promising results in predicting type of DTI, which was achieved via integrating and mapping triplet drug–target–disease association graphs into low-dimensional dense vectors. To our knowledge, this is the first approach that addresses prediction between drugs and targets across six interaction types.

Distributed non-negative RESCAL with automatic model selection for exascale data

With the boom in the development of computer hardware and software, social media, IoT platforms, and communications, there has been exponential growth in the volume of data produced worldwide. Among these data, relational datasets are growing in popularity as they provide unique insights regarding the evolution of communities and their interactions. Relational datasets are naturally non-negative, sparse, and extra-large. Relational data usually contain triples (subject, relation, object) and are represented as graphs/multigraphs, called knowledge graphs, which need to be embedded into a low-dimensional dense vector space. Among various embedding models, RESCAL allows the learning of relational data to extract the posterior distributions over the latent variables and to make predictions of missing relations. However, RESCAL is computationally demanding and requires a fast and distributed implementation to analyze extra-large real-world datasets. Here we introduce a distributed non-negative RESCAL algorithm for heterogeneous CPU/GPU architectures with automatic selection of the number of latent communities (model selection), called pyDRESCALk. We demonstrate the correctness of pyDRESCALk with real-world and large synthetic tensors and the efficacy showing near-linear scaling that concurs with the theoretical complexities. Finally, pyDRESCALk determines the number of latent communities in an 11-terabyte dense and 9-exabyte sparse synthetic tensor.

Computer-aided diagnosis of schizophrenia based on node2vec and Transformer

Objective:Compared with the healthy control (HC) group, the brain structure and function of schizophrenia (SZ) patients are significantly abnormal, so brain imaging methods can be used to achieve the aided diagnosis of SZ. However, a brain network based on brain imaging data is non-Euclidean, and its intrinsic features cannot be learned effectively by general deep learning models. Furthermore, in the majority of existing studies, brain network features were manually specified as the input of machine learning models. Methods:In this study, brain functional network constructed from the subject’s fMRI data is analyzed, and its small-world value is calculated and t-tested; the node2vec algorithm in graph embedding is introduced to transform the constructed brain network into low-dimensional dense vectors, and the brain network’s non-Euclidean spatial structure characteristics are retained to the greatest extent, so that its intrinsic features can be extracted by deep learning models; GridMask is used to randomly mask part of the information in the vectors to enhance the data; and then features can be extracted using the Transformer model to identify SZ. Results:It is again shown that the small-world value of the brain network in SZ is significantly lower than that in HC by t-test (p=0.014¡0.05). 97.78% classification accuracy is achieved by the proposed methods (node2vec + GridMask + Transformer) in 30 SZ patients and 30 healthy people. Conclusion:The experiment shows that the node2vec used in this paper can effectively solve the problem of brain network features being difficult to learn by general deep learning models. The high-precision computer-aided diagnosis of SZ can be obtained by combining node2vec with Transformer and GridMask. Significance:The proposed methods in the paper are expected to be used for aided diagnosis of SZ.

Improved DeepFM Recommendation Algorithm Incorporating Deep Feature Extraction

In recent years, deep learning has been applied to the field of recommendation, which can learn complex user interaction features and make better recommendations. However, deep learning only focuses on the interaction of high-order features and neglects the low-order features. The DeepFM model combines the linear FM (Factorization Machines) model and the deep DNN (Deep Neural Network) model to realize the interactive learning of low-order and high-order features, but it does not take into account that user interests will change dynamically with time. When the data sparsity is high, it cannot be effectively recommended. Based on this, an improved DeepFM recommendation algorithm that combines depth feature extraction was proposed, named fDeepFM. Firstly, the word features are transformed into low-dimensional dense vectors through the Embedding layer. Then Doc2Vec is combined to mine item features with context, and the two are stitched together as the input to the FM model and DNN model. Subsequently, user features are input to the GRU (Gated Cyclic Unit) model according to different cycles to mine user features. Finally, the results of the FM model, DNN model, and GRU model are combined by linear stitching as the overall output of the fDeepFM model. Experiments were carried out on Movielens-20M and Amazon data sets. The experimental results showed that MAE, RMSE, F1-score, and AUC on the Movielens-20M data set were optimized by 1.69%, 2.4%, 1.67%, and 2.28%, respectively; On the Amazon dataset, MAE, RMSE, F1-score, and AUC are optimized by 3.2%, 3.86%, 1.63%, and 2.2% respectively compared with DeepFM.

A Sustainable Solution for IoT Semantic Interoperability: Dataspaces Model via Distributed Approaches

In the past few decades, the prevalence of Internet of Things (IoT) applications has brought both opportunities and challenges of different categories. One of the main concerns nowadays is semantic interoperability. Although plenty of augmented ontologies and resource description frameworks have been designed to achieve semantic interoperation, there is still no sustainable solution to interconnect the increasing number of heterogeneous devices and corresponding data. Aiming at this problem, this article presents a dataspaces model utilizing distributed approaches to represent semantic information as a sustainable solution for IoT semantic interoperability. In this model, an attention-based entity embedding approach is designed to convert IoT entities into low-dimensional dense vectors, then calculations, including entities, relations, etc., can be further conducted. Consequently, the entity embeddings could reflect the semantic information of Class equivalences among entities. To recognize entity relations, the generated entity vectors and the arithmetic results of two entities are combined as input features. By feeding the input features to a well-trained Tensor-train modified DNN, the relation between two entities could be recognized. Finally, experiments are conducted on two data sets to evaluate the proposed approaches. Results indicate that the generated entity vectors could effectively reflect the semantic similarity between entities. More importantly, while achieving parameter compression and better generalizing ability, the relation recognition approach improves the relation recognition accuracy on new entities compared with the state-of-art models.

Novel drug-target interactions via link prediction and network embedding

BackgroundAs many interactions between the chemical and genomic space remain undiscovered, computational methods able to identify potential drug-target interactions (DTIs) are employed to accelerate drug discovery and reduce the required cost. Predicting new DTIs can leverage drug repurposing by identifying new targets for approved drugs. However, developing an accurate computational framework that can efficiently incorporate chemical and genomic spaces remains extremely demanding. A key issue is that most DTI predictions suffer from the lack of experimentally validated negative interactions or limited availability of target 3D structures.ResultsWe report DT2Vec, a pipeline for DTI prediction based on graph embedding and gradient boosted tree classification. It maps drug-drug and protein–protein similarity networks to low-dimensional features and the DTI prediction is formulated as binary classification based on a strategy of concatenating the drug and target embedding vectors as input features. DT2Vec was compared with three top-performing graph similarity-based algorithms on a standard benchmark dataset and achieved competitive results. In order to explore credible novel DTIs, the model was applied to data from the ChEMBL repository that contain experimentally validated positive and negative interactions which yield a strong predictive model. Then, the developed model was applied to all possible unknown DTIs to predict new interactions. The applicability of DT2Vec as an effective method for drug repurposing is discussed through case studies and evaluation of some novel DTI predictions is undertaken using molecular docking.ConclusionsThe proposed method was able to integrate and map chemical and genomic space into low-dimensional dense vectors and showed promising results in predicting novel DTIs.

Network Embedding Algorithm Taking in Variational Graph AutoEncoder

Complex networks with node attribute information are employed to represent complex relationships between objects. Research of attributed network embedding fuses the topology and the node attribute information of the attributed network in the common latent representation space, to encode the high-dimensional sparse network information to the low-dimensional dense vector representation, effectively improving the performance of the network analysis tasks. The current research on attributed network embedding is presently facing problems of high-dimensional sparsity of attribute eigenmatrix and underutilization of attribute information. In this paper, we propose a network embedding algorithm taking in a variational graph autoencoder (NEAT-VGA). This algorithm first pre-processes the attribute features, i.e., the attribute feature learning of the network nodes. Then, the feature learning matrix and the adjacency matrix of the network are fed into the variational graph autoencoder algorithm to obtain the Gaussian distribution of the potential vectors, which more easily generate high-quality node embedding representation vectors. Then, the embedding of the nodes obtained by sampling this Gaussian distribution is reconstructed with structural and attribute losses. The loss function is minimized by iterative training until the low-dimension vector representation, containing network structure information and attribute information of nodes, can be better obtained, and the performance of the algorithm is evaluated by link prediction experimental results.

Drug-target interactions prediction via deep collaborative filtering with multiembeddings.

Drug-target interactions (DTIs) prediction research presents important significance for promoting the development of modern medicine and pharmacology. Traditional biochemical experiments for DTIs prediction confront the challenges including long time period, high cost and high failure rate, and finally leading to a low-drug productivity. Chemogenomic-based computational methods can realize high-throughput prediction. In this study, we develop a deep collaborative filtering prediction model with multiembeddings, named DCFME (deep collaborative filtering prediction model with multiembeddings), which can jointly utilize multiple feature information from multiembeddings. Two different representation learning algorithms are first employed to extract heterogeneous network features. DCFME uses the generated low-dimensional dense vectors as input, and then simulates the drug-target relationship from the perspective of both couplings and heterogeneity. In addition, the model employs focal loss that concentrates the loss on sparse and hard samples in the training process. Comparative experiments with five baseline methods show that DCFME achieves more significant performance improvement on sparse datasets. Moreover, the model has better robustness and generalization capacity under several harder prediction scenarios.

Electronic Medical Records as Input to Predict Postoperative Immediate Remission of Cushing’s Disease: Application of Word Embedding

BackgroundNo existing machine learning (ML)-based models use free text from electronic medical records (EMR) as input to predict immediate remission (IR) of Cushing’s disease (CD) after transsphenoidal surgery.PurposeThe aim of the present study is to develop an ML-based model that uses EMR that include both structured features and free text as input to preoperatively predict IR after transsphenoidal surgery.MethodsA total of 419 patients with CD from Peking Union Medical College Hospital were enrolled between January 2014 and August 2020. The EMR of the patients were embedded and transformed into low-dimensional dense vectors that can be included in four ML-based models together with structured features. The area under the curve (AUC) of receiver operating characteristic curves was used to evaluate the performance of the models.ResultsThe overall remission rate of the 419 patients was 75.7%. From the results of logistic multivariate analysis, operation (p < 0.001), invasion of cavernous sinus from MRI (p = 0.046), and ACTH (p = 0.024) were strongly correlated with IR. The AUC values for the four ML-based models ranged from 0.686 to 0.793. The highest AUC value (0.793) was for logistic regression when 11 structured features and “individual conclusions of the case by doctor” were included.ConclusionAn ML-based model was developed using both structured and unstructured features (after being processed using a word embedding method) as input to preoperatively predict postoperative IR.

Multilayer Network Representation Learning Method Based on Random Walk of Multiple Information

Network representation learning aims to map nodes in the network into low-dimensional dense vectors, which can be widely used to solve the network analysis tasks. Existing methods mainly focus on single-layer homogeneous networks. However, many real-world networks consist of multiple types of nodes and edges, which are called multilayer networks. The problem of how to capture node information and use multi-type relational information is a major challenge of multilayer network representation learning. To address this problem, we propose a method of random walk of multiple information, called IFMNE, to efficiently preserve and learn node information and multi-type relational information into a unified space. This method combines node structure information with network topology information to obtain the node random walk sequence, and trains the node walk sequence on the neural network model. Experimental results are performed on five real multilayer networks, and the embedding vectors were evaluated by link prediction task. The accuracy was significantly improved on the basis of low time complexity compared with the baseline methods.

Internet Financial Fraud Detection Based on a Distributed Big Data Approach With Node2vec

The rapid development of information technologies like Internet of Things, Big Data, Artificial Intelligence, Blockchain, etc., has profoundly affected people's consumption behaviors and changed the development model of the financial industry. The financial services on Internet and IoT with new technologies has provided convenience and efficiency for consumers, but new hidden fraud risks are generated also. Fraud, arbitrage, vicious collection, etc., have caused bad effects and huge losses to the development of finance on Internet and IoT. However, as the scale of financial data continues to increase dramatically, it is more and more difficult for existing rule-based expert systems and traditional machine learning model systems to detect financial frauds from large-scale historical data. In the meantime, as the degree of specialization of financial fraud continues to increase, fraudsters can evade fraud detection by frequently changing their fraud methods. In this article, an intelligent and distributed Big Data approach for Internet financial fraud detections is proposed to implement graph embedding algorithm Node2Vec to learn and represent the topological features in the financial network graph into low-dimensional dense vectors, so as to intelligently and efficiently classify and predict the data samples of the large-scale dataset with the deep neural network. The approach is distributedly performed on the clusters of Apache Spark GraphX and Hadoop to process the large dataset in parallel. The groups of experimental results demonstrate that the proposed approach can improve the efficiency of Internet financial fraud detections with better precision rate, recall rate, F1-Score and F2-Score.

Exploring Temporal Information for Dynamic Network Embedding

Representing nodes in a network as low-dimensional dense vectors can facilitate the analysis of complex networks, which is a challenging task and has attracted increasing attention. However, in the real world, networks are changing over time, such as cooperation in citation networks and communication in email networks. Most of the recent embedding methods only focus on static networks. Thus they ignore the critical temporal information, which serves as a supplement to structure information and has been proved to improve the quality of node embedding. In this work, we propose an unsupervised deep learning model called DTINE, which explores temporal information for further enhancing the robustness of node representations in dynamic networks. To preserve network topology, we pertinently design a temporal weight and sampling strategy to extract features from the neighborhoods. An attention mechanism will be applied on the recurrent neural network to measure the contributions of historical information and capture the evolution of the networks. Experimental results on four real-world networks demonstrate that the proposed method achieves better performance than state-of-the-art methods.

A neural collaborative filtering method for identifying miRNA-disease associations

The identification of disease-associated miRNAs can help people better understand the pathogenesis of diseases from a genetic perspective. Therefore, the prediction of miRNA-disease associations has received increasing attention. In this paper, we propose a new computational method NCFM (Neural network-based Collaborative Filtering Method) to predict miRNA-disease associations based on deep neural network. Firstly, high-dimensional sparse vectors of diseases and miRNAs are mapped into low-dimensional dense vectors in implicit semantic space via embedding layer, which called disease embedding and miRNA embedding, respectively. Secondly, the neural collaborative filter layers model the latent feature interactions between miRNAs and diseases. Then, different from other methods using square error loss function, we propose a new pairwise loss function to optimizes our model from a ranking perspective. Finally, experiments show that our proposed method can effectively prioritize disease-related miRNAs with the highest AUC of 0.912 and 0.921 compared with other recent methods in 5-fold cross validation and LOOCV framework. In addition, we implement two types of case studies, including four diseases. For a disease, more than 90% of predicted miRNAs are validated by another official dataset, which further illustrates the effectiveness of NCFM.

DMFMDA: Prediction of Microbe-Disease Associations Based on Deep Matrix Factorization Using Bayesian Personalized Ranking.

Identifying the microbe-disease associations is conducive to understanding the pathogenesis of disease from the perspective of microbe. In this paper, we propose a deep matrix factorization prediction model (DMFMDA) based on deep neural network. First, the disease one-hot encoding is fed into neural network, which is transformed into a low-dimensional dense vector in implicit semantic space via embedding layer, and so is microbe. Then, matrix factorization is realized by neural network with embedding layer. Furthermore, our model synthesizes the non-linear modeling advantages of multi-layer perceptron based on the linear modeling advantages of matrix factorization. Finally, different from other methods using square error loss function, Bayesian Personalized Ranking optimizes the model from a ranking perspective to obtain the optimal model parameters, which makes full use of the unobserved data. Experiments show that DMFMDA reaches average AUCs of 0.9091 and 0.9103 in the framework of 5-fold cross validation and Leave-one-out cross validation, which is superior to three the-state-of-art methods. In case studies, 10, 9 and 9 out of top-10 candidate microbes are verified by recently published literature for asthma, inflammatory bowel disease and colon cancer, respectively. In conclusion, DMFMDA is successful application of deep learning in the prediction of microbe-disease association.