Continuous Vector Representations Research Articles

Classification of Depression Expressions on Twitter Using Ensemble Learning with Word2Vec

One of the mental health disorders experienced by people is depression. Depression is a mental disorder characterized by feelings of sadness, loss of interest or pleasure in daily activities, and decreased cognitive function that can affect social life, work, and general physical health. Early detection is needed to prevent the occurrence of bad risks. One of the early detections can be done through social media. This is because social media is one of the tools used to channel expression. This research uses data taken from Twitter social media to create a machine learning model. Before model building, data pre-processing will be carried out using Word2Vec to convert text into a continuous vector representation. The algorithm used is ensemble learning by combining five algorithms: Logistic Regression, Decision Tree, K-Nearest Neighbor, Artificial Neural Network, and Support Vector Machine. The results show that different Word2Vec architectures can give the model another performance. Ensemble Learning can improve the performance of using a single model. The best results were obtained by using a data ratio of 90:10 using the Skip-gram architecture to get an accuracy value and f1-score of 94%.

Kmer-Node2Vec: a Fast and Efficient Method for Kmer Embedding from the Kmer Co-occurrence Graph, with Applications to DNA Sequences.

Learning low-dimensional continuous vector representation for short k-mers divided from long DNA sequences is key to DNA sequence modeling that can be utilized in many bioinformatics investigations, such as DNA sequence retrieval and classification. DNA2Vec is the most widely used method for DNA sequence embedding. However, it poorly scales to large data sets due to its extremely long training time in kmer embedding. In this paper, we propose a novel efficient graph-based kmer embedding method, named Kmer-Node2Vec, to tackle this concern. Our method converts the large DNA corpus into one kmer co-occurrence graph, and extracts kmer relation on the graph by random walks to learn fast and high-quality kmer embedding. Extensive experiments show that our method is faster than DNA2Vec by 29 times for training on a 4GB data set, and on par with DNA2Vec in terms of task-specific accuracy of sequence retrieval and classification.

Combining discrete choice models and neural networks through embeddings: Formulation, interpretability and performance

This paper proposes a novel approach that combines theory- and data-driven choice models using Artificial Neural Networks (ANNs). In particular, we use continuous vector representations, called embeddings, for encoding categorical or discrete explanatory variables with a special focus on interpretability and model transparency. Although embedding representations within the logit framework have been conceptualized by Pereira (2019), their dimensions do not have an absolute definitive meaning, hence offering limited behavioral insights in this earlier work. The novelty of our work lies in enforcing interpretability to the embedding vectors by formally associating each of their dimensions to a choice alternative. Thus, our approach brings benefits much beyond a simple parsimonious representation improvement over dummy encoding, as it provides behaviorally meaningful outputs that can be used in travel demand analysis and policy decisions. Additionally, in contrast to previously suggested ANN-based Discrete Choice Models (DCMs) that either sacrifice interpretability for performance or are only partially interpretable, our main model (E-MNL) preserves interpretability of the utility coefficients for all the input variables despite being based on ANN principles. The proposed models are tested on two real world datasets and evaluated against benchmark and baseline models that use dummy-encoding. The results of the experiments indicate that our models deliver state-of-the-art predictive performance, outperforming existing ANN-based models while drastically reducing the number of required network parameters.

A multi-scale graph embedding method via multiple corpora

Graph embedding aims at learning continuous vector representations for graphs which is crucial for graph analytics. Natural Language Process (NLP)-based graph embedding methods build corpus for graph data by treating substructures as words and then use NLP models to learn graph embeddings. However, the size difference and data redundancy among substructures are less explored in the built corpora. To mitigate this problem, we propose an unsupervised multi-scale graph embedding method. To be a specific, we first build multiple graph corpora for a graph dataset, where each corpus only contains substructures of specific granularity. Then, we extend a document embedding model to each graph corpus to obtain graph embeddings of different scales. At last, we obtain the final multi-scale embedding of a graph by pooling its multiple embeddings. Comprehensive experiments on real graph datasets indicate that the proposed method obtains competitive results with state-of-the-arts, and is superior to some classic graph kernels and graph embedding methods on six out of ten benchmark datasets.

Inverse design of viral infectivity-enhancing peptide fibrils from continuous protein-vector embeddings.

Amyloid-like nanofibers from self-assembling peptides can promote viral gene transfer for therapeutic applications. Traditionally, new sequences are discovered either from screening large libraries or by creating derivatives of known active peptides. However, the discovery of de novo peptides, which are sequence-wise not related to any known active peptides, is limited by the difficulty to rationally predict structure-activity relationships because their activities typically have multi-scale and multi-parameter dependencies. Here, we used a small library of 163 peptides as a training set to predict de novo sequences for viral infectivity enhancement using a machine learning (ML) approach based on natural language processing. Specifically, we trained an ML model using continuous vector representations of the peptides, which were previously shown to retain relevant information embedded in the sequences. We used the trained ML model to sample the sequence space of peptides with 6 amino acids to identify promising candidates. These 6-mers were then further screened for charge and aggregation propensity. The resulting 16 new 6-mers were tested and found to be active with a 25% hit rate. Strikingly, these de novo sequences are the shortest active peptides for infectivity enhancement reported so far and show no sequence relation to the training set. Moreover, by screening the sequence space, we discovered the first hydrophobic peptide fibrils with a moderately negative surface charge that can enhance infectivity. Hence, this ML strategy is a time- and cost-efficient way for expanding the sequence space of short functional self-assembling peptides exemplified for therapeutic viral gene delivery.

TissueSpace: a web tool for rank-based transcriptome representation and its applications in molecular medicine.

Cross-platform or cross-experiment transcriptome data is hard to compare as the original gene expression values from different platforms cannot be compared directly. The inherent gene expression ranking information is rarely utilized. Use of reduced vector to represent transcriptome data independent of platforms. Thus, we turned the expression profile into a rank vector, where a higher expression has a higher rank value, then applied Latent semantic analysis (LSA) to get compact and continuous 100-dimensional vector representations for samples. Results showed that the reconstructed vector has a precision of 96.7% in recovering tissue labels from an independent dataset. A user-friendly tool TissueSpace was developed, which provides users the following functionalities: (1) convert different gene ID types to Ensembl gene IDs; (2) project any human transcriptome profile to get vector representation for downstream analysis; (3) functional enrichment for each of the 100-dimensional vector features. Case studies for its applications in human common diseases indicate its usefulness. TissueSpace could be used to generate testable hypotheses for translational medicine. The TissueSpace tool is available at http://bioinformatics.fafu.edu.cn/tissuespace/ .

A Survey on Deep Learning for Named Entity Recognition

Named entity recognition (NER) is the task to identify mentions of rigid designators from text belonging to predefined semantic types such as person, location, organization etc. NER always serves as the foundation for many natural language applications such as question answering, text summarization, and machine translation. Early NER systems got a huge success in achieving good performance with the cost of human engineering in designing domain-specific features and rules. In recent years, deep learning, empowered by continuous real-valued vector representations and semantic composition through nonlinear processing, has been employed in NER systems, yielding stat-of-the-art performance. In this paper, we provide a comprehensive review on existing deep learning techniques for NER. We first introduce NER resources, including tagged NER corpora and off-the-shelf NER tools. Then, we systematically categorize existing works based on a taxonomy along three axes: distributed representations for input, context encoder, and tag decoder. Next, we survey the most representative methods for recent applied techniques of deep learning in new NER problem settings and applications. Finally, we present readers with the challenges faced by NER systems and outline future directions in this area.

Task-oriented attributed network embedding by multi-view features

Network embedding, also known as network representation learning, aims at defining low-dimensional, continuous vector representation of nodes to maximally preserve the network structure. Recent efforts attempt to extend network embedding to attributed networks where nodes are enriched with descriptors, to enhance interpretability. However, most of these efforts seldom consider the additional knowledge relevant to the aim of the downstream network analysis, i.e. task-related information. When they do, they are analysis-specific and thus lack adaptability to alternative tasks. In this article, a unified framework TANE is proposed to learn Task-oriented Attributed Network Embedding that jointly, maximally and consistently preserves multiple types of network information to generate rich nodes representations, robust to a variety of analyses. The framework can flexibly adapt to, and be readily modified for, different network-based tasks in an end-to-end way. The results of extensive experiments on well-known and commonly used datasets demonstrate that the proposed framework TANE can achieve superior performance over state-of-the-art methods in two commonly performed tasks: node classification and link prediction.

Graph embedding via multi-scale graph representations

Graph embedding provides an effective way to encode graph nodes as continuous vector representations in a low-dimensional space. The high-order proximities based graph embedding methods can preserve global structures, but the high-order proximities based objectives typically imply non-convex optimization and high computational complexity. In contrast, the low-order proximities based graph embedding methods evade the definition and optimization of complex high-order proximities based objectives, but cannot capture global structures. Furthermore, numerous graph embedding methods that only consider the proximity relationships among nodes ignore to capture global structure from the perspective of subgraphs. Motivated by this, we propose a novel graph embedding framework via multi-scale graph representations, named MSGE. MSGE first determines multiple subgraphs of different scales based on random walk. At each scale, MSGE generates a new graph by creating edges among the subgraphs which are treated as supernodes. Thus, the generated multi-scale graphs can approximate rich global structure of the original graph. MSGE then employs any existing graph embedding method as a black box to learn subgraph (i.e., supernode) embeddings on each generated graph. Subsequently, a first-order proximity based embedding fusion method is devised to yield node embeddings of the original graph via the learned multi-scale subgraph embeddings. Finally, we apply MSGE on four classical graph embedding methods and extensive experimental results demonstrate that our framework can generate embeddings of better quality and significantly outperform the original graph embedding methods on visualization, node classification and community detection tasks.

A network embedding-enhanced Bayesian model for generalized community detection in complex networks

Community detection has been a significant and long-standing task in network analysis. While existing methods tend to focus on the assortative structure where communities are densely connected internally, in reality other kinds of network structure (e.g., disassortative) also exist. In addition, previous methods often have difficulties in dealing with noise and redundant information in the network topology. To address these problems, we develop a novel Bayesian probabilistic model for identifying the generalized communities, regardless of whether the network structure is assortative, disassortative, or otherwise. Specifically, the model combines the original adjacency matrix representation with the network embedding (the dense and continuous vector representations of nodes in a low-dimensional space). The two parts are connected and generated jointly via the community memberships of nodes learned from the model. Finally, we take a Bayesian treatment for model parameters and develop an efficient variational inference algorithm to detect communities. Experimental results demonstrate the outstanding performance of the new approach both on synthetic networks and on real-world networks, while case studies validate the ability of the proposed approach to describe generalized communities meaningfully.

Semantic Table Retrieval Using Keyword and Table Queries

Tables on the Web contain a vast amount of knowledge in a structured form. To tap into this valuable resource, we address the problem of table retrieval: answering an information need with a ranked list of tables. We investigate this problem in two different variants, based on how the information need is expressed: as a keyword query or as an existing table (“query-by-table”). The main novel contribution of this work is a semantic table retrieval framework for matching information needs (keyword or table queries) against tables. Specifically, we (i) represent queries and tables in multiple semantic spaces (both discrete sparse and continuous dense vector representations) and (ii) introduce various similarity measures for matching those semantic representations. We consider all possible combinations of semantic representations and similarity measures and use these as features in a supervised learning model. Using two purpose-built test collections based on Wikipedia tables, we demonstrate significant and substantial improvements over state-of-the-art baselines.

Search Efficient Binary Network Embedding

Traditional network embedding primarily focuses on learning a continuous vector representation for each node, preserving network structure and/or node content information, such that off-the-shelf machine learning algorithms can be easily applied to the vector-format node representations for network analysis. However, the learned continuous vector representations are inefficient for large-scale similarity search, which often involves finding nearest neighbors measured by distance or similarity in a continuous vector space. In this article, we propose a search efficient binary network embedding algorithm called BinaryNE to learn a binary code for each node, by simultaneously modeling node context relations and node attribute relations through a three-layer neural network. BinaryNE learns binary node representations using a stochastic gradient descent-based online learning algorithm. The learned binary encoding not only reduces memory usage to represent each node, but also allows fast bit-wise comparisons to support faster node similarity search than using Euclidean or other distance measures. Extensive experiments and comparisons demonstrate that BinaryNE not only delivers more than 25 times faster search speed, but also provides comparable or better search quality than traditional continuous vector based network embedding methods. The binary codes learned by BinaryNE also render competitive performance on node classification and node clustering tasks. The source code of the BinaryNE algorithm is available at https://github.com/daokunzhang/BinaryNE.

Neural embeddings of scholarly periodicals reveal complex disciplinary organizations.

Understanding the structure of knowledge domains is one of the foundational challenges in the science of science. Here, we propose a neural embedding technique that leverages the information contained in the citation network to obtain continuous vector representations of scientific periodicals. We demonstrate that our periodical embeddings encode nuanced relationships between periodicals and the complex disciplinary and interdisciplinary structure of science, allowing us to make cross-disciplinary analogies between periodicals. Furthermore, we show that the embeddings capture meaningful "axes" that encompass knowledge domains, such as an axis from "soft" to "hard" sciences or from "social" to "biological" sciences, which allow us to quantitatively ground periodicals on a given dimension. By offering novel quantification in the science of science, our framework may, in turn, facilitate the study of how knowledge is created and organized.

Explainable automated coding of clinical notes using hierarchical label-wise attention networks and label embedding initialisation

BackgroundDiagnostic or procedural coding of clinical notes aims to derive a coded summary of disease-related information about patients. Such coding is usually done manually in hospitals but could potentially be automated to improve the efficiency and accuracy of medical coding. Recent studies on deep learning for automated medical coding achieved promising performances. However, the explainability of these models is usually poor, preventing them to be used confidently in supporting clinical practice. Another limitation is that these models mostly assume independence among labels, ignoring the complex correlations among medical codes which can potentially be exploited to improve the performance. MethodsTo address the issues of model explainability and label correlations, we propose a Hierarchical Label-wise Attention Network (HLAN), which aimed to interpret the model by quantifying importance (as attention weights) of words and sentences related to each of the labels. Secondly, we propose to enhance the major deep learning models with a label embedding (LE) initialisation approach, which learns a dense, continuous vector representation and then injects the representation into the final layers and the label-wise attention layers in the models. We evaluated the methods using three settings on the MIMIC-III discharge summaries: full codes, top-50 codes, and the UK NHS (National Health Service) COVID-19 (Coronavirus disease 2019) shielding codes. Experiments were conducted to compare the HLAN model and label embedding initialisation to the state-of-the-art neural network based methods, including variants of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). ResultsHLAN achieved the best Micro-level AUC and F1 on the top-50 code prediction, 91.9% and 64.1%, respectively; and comparable results on the NHS COVID-19 shielding code prediction to other models: around 97% Micro-level AUC. More importantly, in the analysis of model explanations, by highlighting the most salient words and sentences for each label, HLAN showed more meaningful and comprehensive model interpretation compared to the CNN-based models and its downgraded baselines, HAN and HA-GRU. Label embedding (LE) initialisation significantly boosted the previous state-of-the-art model, CNN with attention mechanisms, on the full code prediction to 52.5% Micro-level F1. The analysis of the layers initialised with label embeddings further explains the effect of this initialisation approach. The source code of the implementation and the results are openly available at https://github.com/acadTags/Explainable-Automated-Medical-Coding. ConclusionWe draw the conclusion from the evaluation results and analyses. First, with hierarchical label-wise attention mechanisms, HLAN can provide better or comparable results for automated coding to the state-of-the-art, CNN-based models. Second, HLAN can provide more comprehensive explanations for each label by highlighting key words and sentences in the discharge summaries, compared to the n-grams in the CNN-based models and the downgraded baselines, HAN and HA-GRU. Third, the performance of deep learning based multi-label classification for automated coding can be consistently boosted by initialising label embeddings that captures the correlations among labels. We further discuss the advantages and drawbacks of the overall method regarding its potential to be deployed to a hospital and suggest areas for future studies.

A study of continuous vector representations for theorem proving

Abstract Applying machine learning to mathematical terms and formulas requires a suitable representation of formulas that is adequate for AI methods. In this paper, we develop an encoding that allows for logical properties to be preserved and is additionally reversible. This means that the tree shape of a formula including all symbols can be reconstructed from the dense vector representation. We do that by training two decoders: one that extracts the top symbol of the tree and one that extracts embedding vectors of subtrees. The syntactic and semantic logical properties that we aim to preserve include both structural formula properties, applicability of natural deduction steps and even more complex operations like unifiability. We propose datasets that can be used to train these syntactic and semantic properties. We evaluate the viability of the developed encoding across the proposed datasets as well as for the practical theorem proving problem of premise selection in the Mizar corpus.

Microblog sentiment analysis via embedding social contexts into an attentive LSTM

With the rise of microblogging services like Twitter and Sina Weibo, users are able to post various contents on breaking news, public events, or products conveniently and swiftly. These massive contents carry users’ mass sentiment and opinions on various topics, which are a kind of useful and timely source. Traditional microblog sentiment analysis methods often assume that microblogs are independent and identically distributed, they ignore the fact that the microblogs are networked data. Although some methods take the relations between microblogs into consideration, they only use shallow network features which are not sufficient, such as neighbors. Besides, these methods are content-based methods because they cannot use social context information in the prediction stage. To solve this problem, in this paper we use a deep learning method to fully capture the features of microblog relations including both the implicit and explicit ones and use these features to promote microblog sentiment analysis results. Specifically, we first construct a graph which models the relations between microblogs inspired by sentiment consistency and emotional contagion theories. Then we embed the microblog graph and get a continuous vector representation for social contexts of each microblog. After that, we propose a novel neural network to integrate social context knowledge with text information. To handle the problem that different words have different contributions to the classification result, we introduce the attention mechanism into our model. We conduct experiments on three publicly released datasets. The experimental results show that our proposed model can outperform state-of-the-art methods consistently and significantly.

Reducing the complexity of financial networks using network embeddings

Accounting scandals like Enron (2001) and Petrobas (2014) remind us that untrustworthy financial information has an adverse effect on the stability of the economy and can ultimately be a source of systemic risk. This financial information is derived from processes and their related monetary flows within a business. But as the flows are becoming larger and more complex, it becomes increasingly difficult to distill the primary processes for large amounts of transaction data. However, by extracting the primary processes we will be able to detect possible inconsistencies in the information efficiently. We use recent advances in network embedding techniques that have demonstrated promising results regarding node classification problems in domains like biology and sociology. We learned a useful continuous vector representation of the nodes in the network which can be used for the clustering task, such that the clusters represent the meaningful primary processes. The results show that we can extract the relevant primary processes which are similar to the created clusters by a financial expert. Moreover, we construct better predictive models using the flows from the extracted primary processes which can be used to detect inconsistencies. Our work will pave the way towards a more modern technology and data-driven financial audit discipline.

An Attributed Network Representation Learning Method Based on Biased Random Walk

Network representation learning(NRL) aims to learn the low-dimensional and continuous vector representations for all nodes in networks, which is used as the input feature for many complex networks analysis tasks. Random walk has a wide range of applications in structure-based network representation learning. In reality, the network contains a lot of attribute information, which brings opportunities and challenges to random walk. Combining attribute information can reconstruct the network better. Based on the above, we propose a robust NRL method called Attributed Network Representation Learning Based on Biased Random Walk (ANRLBRW). This method considers the structure and homogeneity of the network through a biased random walk. It uses the attribute random walk to take care of the node’s attribute information, and combines two random walk sequences to learn the network representation of the node using the skip-gram model. The experimental results on three real attributed networks show that the accuracy of node classification outperforms the state-of-the-art methods on the basis of our proposed network representation learning method.

A Multi-Layer Dual Attention Deep Learning Model With Refined Word Embeddings for Aspect-Based Sentiment Analysis

Although the sentiment analysis domain has been deeply studied in the last few years, the analysis of social media content is still a challenging task due to the exponential growth of multimedia content. Natural language ambiguities and indirect sentiments within the social media text have made it hard to classify. Aspect-based sentiment analysis creates a need to develop explicit extraction techniques using syntactic parsers to exploit the relationship between the aspect and sentiment terms. Along with the extraction approaches, word embeddings are generated through Word2Vec models for the continuous low-dimensional vector representation of text that fails to capture the significant sentiment information. This paper presents a co-extraction model with refined word embeddings to exploit the dependency structures without using syntactic parsers. For this purpose, a deep learning-based multilayer dual-attention model is proposed to exploit the indirect relation between the aspect and opinion terms. In addition, word embeddings are refined by providing distinct vector representations to dissimilar sentiments, unlike the Word2Vec model. For this, we have employed a sentiment refinement technique for pre-trained word embedding model to overcome the problem of similar vector representations of opposite sentiments. Performance of the proposed model is evaluated on three benchmark datasets of SemEval Challenge 2014 and 2015. The experimental results indicate the effectiveness of our model compared to the existing state-of-the-art models for the aspect-based sentiment analysis.

Refining Word Embeddings Using Intensity Scores for Sentiment Analysis

Word embeddings that provide continuous low-dimensional vector representations of words have been extensively used for various natural language processing tasks. However, existing context-based word embeddings such as Word2vec and GloVe typically fail to capture sufficient sentiment information, which may result in words with similar vector representations having an opposite sentiment polarity (e.g., good and bad ), thus degrading sentiment analysis performance. To tackle this problem, recent studies have suggested learning sentiment embeddings to incorporate the sentiment polarity (positive and negative) information from labeled corpora. This study adopts another strategy to learn sentiment embeddings. Instead of creating a new word embedding from labeled corpora, we propose a word vector refinement model to refine existing pretrained word vectors using real-valued sentiment intensity scores provided by sentiment lexicons. The idea of the refinement model is to improve each word vector such that it can be closer in the lexicon to both semantically and sentimentally similar words (i.e., those with similar intensity scores) and further away from sentimentally dissimilar words (i.e., those with dissimilar intensity scores). An obvious advantage of the proposed method is that it can be applied to any pretrained word embeddings. In addition, the intensity scores can provide more fine-grained (real-valued) sentiment information than binary polarity labels to guide the refinement process. Experimental results show that the proposed refinement model can improve both conventional word embeddings and previously proposed sentiment embeddings for binary, ternary, and fine-grained sentiment classification on the SemEval and Stanford Sentiment Treebank datasets.