Types Of Embeddings Research Articles

Spectroscopy, Thermodynamics and Molecular Docking of Fraxinellone with DNA.

Fraxinellone is an important botanical lactone compound andhas been demonstrated to have insecticidal activity. To provide theoretical support tothe assessment on the safetyof utilizingfraxinellone as a natural insecticidal agent, the interactions between fraxinellone and armyworm DNA, salmon sperm DNA and calf thymus DNA were investigated using UV-Vis absorption spectroscopy, isothermal titration calorimetry, and molecular docking. Results showed that there were two types of combinations between fraxinellone and three kinds of DNA. Type I combination had an equilibrium constant of combination (Ka1) of about 105 and binding sites (n1) of 0.40-0.70, while type II combination had an equilibrium constant of combination (Ka2) of 103 and binding sites (n2) of 1.35-3.15. Results of molecular docking showed that there werenon-classical embedding type interactions between fraxinellone and three kinds of DNA, with the reaction taking place in small groove areas of the DNA structure, resulting in relatively weak interactive forces.

Alliances of Change Pushing Organizational Transformation Towards Sustainability across 13 Universities

Universities are expected to play a leading role in developing and maintaining sustainability. To contribute to a systemic and dynamic understanding of organizational change that is necessary in order to play such a role, we comparatively analyzed processes of organizational changes towards sustainability across thirteen universities in Austria. This comparative analysis is based on data from guided interviews and document analysis and on validation of preliminary results via group discussion and individual comments. The results show that all universities embedded sustainability in most of their areas of activity (research, teaching, operations, organizational culture, societal engagement), but the depth of integration and the type of structural embedding varies. Especially for early changes dating back to the 1990s, academics working in the broader field of sustainability studies were those agents of change, who—without formal mandate—skillfully and proactively initiated and drove organizational transformations following an idealistic and intrinsic motivation. A timeline analysis illustrates peaks of sustainability-related changes in the years of the foundation of inter-university networks in 2011 and 2017, which acted as alliances of change. Ministry intervention in 2015 helped to bring sustainability on the agenda of those universities with less change agency. In summary, sustainability transformations across the fields of teaching, research, operations, organizational culture, and societal engagement were driven by a fruitful interplay of change agency and change alliances and to a minor degree by top-down interventions.

A Domain-Specific Generative Chatbot Trained from Little Data

Accurate generative chatbots are usually trained on large datasets of question–answer pairs. Despite such datasets not existing for some languages, it does not reduce the need for companies to have chatbot technology in their websites. However, companies usually own small domain-specific datasets (at least in the form of an FAQ) about their products, services, or used technologies. In this research, we seek effective solutions to create generative seq2seq-based chatbots from very small data. Since experiments are carried out in English and morphologically complex Lithuanian languages, we have an opportunity to compare results for languages with very different characteristics. We experimentally explore three encoder–decoder LSTM-based approaches (simple LSTM, stacked LSTM, and BiLSTM), three word embedding types (one-hot encoding, fastText, and BERT embeddings), and five encoder–decoder architectures based on different encoder and decoder vectorization units. Furthermore, all offered approaches are applied to the pre-processed datasets with removed and separated punctuation. The experimental investigation revealed the advantages of the stacked LSTM and BiLSTM encoder architectures and BERT embedding vectorization (especially for the encoder). The best achieved BLUE on English/Lithuanian datasets with removed and separated punctuation was ~0.513/~0.505 and ~0.488/~0.439, respectively. Better results were achieved with the English language, because generating different inflection forms for the morphologically complex Lithuanian is a harder task. The BLUE scores fell into the range defining the quality of the generated answers as good or very good for both languages. This research was performed with very small datasets having little variety in covered topics, which makes this research not only more difficult, but also more interesting. Moreover, to our knowledge, it is the first attempt to train generative chatbots for a morphologically complex language.

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Deep learning methods have revolutionized speech recognition, image recognition, and natural language processing since 2010. Each of these tasks involves a single modality in their input signals. However, many applications in the artificial intelligence field involve multiple modalities. Therefore, it is of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, we provide a technical review of available models and learning methods for multimodal intelligence. The main focus of this review is the combination of vision and natural language modalities, which has become an important topic in both the computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent works on multimodal deep learning from three perspectives: learning multimodal representations, fusing multimodal signals at various levels, and multimodal applications. Regarding multimodal representation learning, we review the key concepts of embedding, which unify multimodal signals into a single vector space and thereby enable cross-modality signal processing. We also review the properties of many types of embeddings that are constructed and learned for general downstream tasks. Regarding multimodal fusion, this review focuses on special architectures for the integration of representations of unimodal signals for a particular task. Regarding applications, selected areas of a broad interest in the current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe that this review will facilitate future studies in the emerging field of multimodal intelligence for related communities.

A critical genre analysis of MD&A discourse in corporate annual reports

Based on the texts of ‘Management Discussion and Analysis on Financial Status and Business’ (MD&A) collected from 118 American corporate annual reports, this study investigated how the professional discourse is realized from the dimensions of text, genre, professional practice and professional culture, according to the framework of the Critical Genre Analysis (CGA) theory. It was found that (1) In the text dimension, lexicon features regarding vocabulary volume, vocabulary highlighting, readability and sentiment are unique to MD&As; (2) the genre of MD&As corresponds to a rhetorical structure of 3 moves and 10 steps; (3) the professional practice of MD&As is represented by three types of interdiscursivity, that is, shifting, mixing and embedding; (4) the professional culture of MD&As is embodied through identity enculturation, human-oriented value, cooperation awareness and self-serving manner. The results of this study help to better understand the realization of professional discourse and expand the application of CGA in professional practice. The findings presented herein are expected to enhance interdisciplinary research that relates language with corporate performance.

Visualizing and exploring POI configurations of urban regions on POI-type semantic space

POI configuration of a region indicates a combination of POI counts on various types inside the region. Exploring POI configurations of urban regions facilitates to understand their functionalities, vibrancy, and developments. However, current studies and applications mainly make statistics toward POI counts on various types separately, neglecting the implicit semantic relations between different types and failing to uncover POI-configuration patterns intuitively. This study proposes a novel framework for visualizing and exploring POIs on POI-type semantic space, with semantic relations of the types being considered. Firstly, using a word-embedding technique (i.e., Word2Vec), the embeddings of POI types are learned from their neighborships on geographic space. Secondly, using a dimension-reduction technique (i.e., t-SNE), all POI-type embeddings are mapped onto 2-dimensional semantic space, where related or similar types would be nearer to each other. Finally, taking the POI-type semantic space as a “base map”, POI configuration of each region is rendered as a “thematic map” by POI counts on various types, which can help intuitively understand and conveniently compare urban regions. The proposed framework can be applied to any urban region and any POI data source, which provides an effective information delivery method for urban planning and POI related urban studies and applications.

A deep learning analysis on question classification task using Word2vec representations

Question classification is a primary essential study for automatic question answering implementations. Linguistic features take a significant role to develop an accurate question classifier. Recently, deep learning systems have achieved remarkable success in various text-mining problems such as sentiment analysis, document classification, spam filtering, document summarization, and web mining. In this study, we explain our study on investigating some deep learning architectures for a question classification task in a highly inflectional language Turkish that is an agglutinative language where word structure is produced by adding suffixes (morphemes) to root word. As a non-Indo-European language, languages like Turkish have some unique features, which make it challenging for natural language processing. For instance, Turkish has no grammatical gender and noun classes. In this study, user questions in Turkish are used to train and test the deep learning architectures. In addition to this, the details of the deep learning architectures are compared in terms of test and 10-cross fold validation accuracy. We use two major deep learning models in our paper: long short-term memory (LSTM), Convolutional Neural Networks (CNN), and we also implemented the combination of CNN-LSTM, CNN-SVM structures and a number of various those architectures by changing vector sizes and the embedding types. As well as this, we have built word embeddings using the Word2vec method with a CBOW and skip gram models with different vector sizes on a large corpus composed of user questions. Our another investigation is the effect of using different Word2vec pre-trained word embeddings on these deep learning architectures. Experiment results show that the use of different Word2vec models has a significant impact on the accuracy rate on different deep learning models. Additionally, there is no Turkish question dataset labeled and so another contribution in this study is that we introduce new Turkish question dataset which is translated from UIUC English question dataset. By using these techniques, we have reached an accuracy of 94% on the question dataset.

Duplicate Questions Pair Detection Using Siamese MaLSTM

Quora is a growing platform comprising a user generated collection of questions and answers. The questions and answers are created, edited, and organized by the users. Enormous number of users on the Quora website makes it unavoidable to have multiple questions from different users with similar intent, which raises the issue of duplicate questions. Effectively detecting duplicate questions would make it easier to find high quality answers and help save time, which in turn would result in an improved user experience for writers and readers on Quora. In this paper, Quora Question Pairs dataset is collected from Kaggle for detection of duplicate questions. First, three types of word embeddings involving Google news vector embedding, FastText crawl embedding with 300 dimensions, and FastText crawl sub words embedding with 300 dimensions are implemented individually to vectorize all the questions and train the model. The final features used for prediction are blend of these three types of word embeddings. Then, Siamese MaLSTM (“Ma” for Manhattan distance) Neural Network model is applied for prediction of duplicate questions in the dataset. Finally, the model is tested on 100000 pairs of questions. The experiments show that the proposed model achieves 91.14% accuracy which is better than the state-of-the-art models.

Data Augmentation Using Deep Generative Models for Embedding Based Speaker Recognition

Data augmentation is an effective method to improve the robustness of embedding based speaker verification systems, which could be applied to either the front-end speaker embedding extractor or the back-end PLDA. Different from the conventional augmentation methods such as manually adding noise or reverberation to the original audios, in this article, we propose to use deep generative models to directly generate more diverse speaker embeddings, which would be used for robust PLDA training. Conditional GAN, and VAE are designed, and investigated for different embedding types, including factor analysis based i-vector, TDNN based x-vector, and ResNet based r-vector. The proposed back-end augmentation methods are evaluated on NIST SRE 2016, and 2018 dataset. Within the popular x-vector, and r-vector framework, the experimental results show that our proposed methods can outperform the traditional audio based back-end augmentation method while different front-end augmentation methods are considered.

Detection of Suicidal Intent in Spanish Language Social Networks using Machine Learning

Suicide is a considerable problem in our population, early intervention for its prevention has a very important role, in order to counteract the number of deaths from suicide. Today, just over half of the world’s population uses social networks, where they express ideas, feelings, desires, including suicide intentions. Motivated by these factors, the main objective is the automatic detection of suicidal ideations in social networks in the Spanish language, in order to serve as a base component to alert and achieve early and specialized interventions. For this, a Spanish suicide phrase classification model has been implemented, since currently no related works in this language with a machine learning approach were found. However, there were some challenges in performing this task, such as understand-ing natural language, generating training data, and obtaining reliable accuracy in classifying these phrases. To construct our classification model, two opposite and popular types of phrase embeddings were chosen, and the most widely used classification algorithms in the literature were compared. Obtaining, as a result, the confirmation that it is possible to classify phrases with suicidal ideation in the Spanish language with good accuracy using semantic representations.

Automatic recognition of messages from virtual communities of drug addicts

The paper describes building a binary classifier with Convolutional Neural Network (CNN) using two different types of word vector representations, Bag-of-Words and Word Embeddings. The purpose of the classifier is to recognise messages published in virtual communities of drug-addicted people. This system may find application in healthcare as a tool for automatic identification of addicts’ communities. It may also provide insights on the features of addicts’ online discourse. The classifier is based on the dataset from Russian-speaking online VK (VKontakte) communities. The dataset comprises texts of publications and comments posted in two types of open communities. The first type includes communities which actively discuss problems of addiction to psychotropic and psychoactive substance. The second type of communities focuses on the discussion of private issues — the users share their life stories and ask for help or advice. In the latter case publications are not related to drug addiction issues. The experiments centered around the development, evaluation and comparative analyses of two models — based on Bag-of-Words and Word Embeddings, respectively. The neural network training was implemented with the Tesla T4 graphics processing unit on the Google Colab platform. The model with the best performance showed 0.99 F1-Score and 0.95 Accuracy; however, after the programme testing, a few weaknesses were found. The programme still requires retraining on a supplemented dataset which includes publications collected from both addicts’ and non-addicts’ communities describing various mental conditions including depression, anxiety and nervous disorders. This opens up an opportunity to create software that can automatically distinguish publications made by people struggling with depression caused by the use of psychoactive substances from publications made by people suffering from depressive disorders of a different kind.

Adversarial Attentive Multi-Modal Embedding Learning for Image-Text Matching

Matching the image and text with deep models has been extensively studied in recent years. Mining the correlation between image and text to learn effective multi-modal features is crucial for image-text matching. However, most existing approaches model the different types of correlation independently. In this work, we propose a novel model named Adversarial Attentive Multi-modal Embedding Learning (AAMEL) for image-text matching. It combines adversarial networks and attention mechanism to learn effective and robust multi-modal embeddings for better matching between the image and text. Adversarial learning is implemented as an interplay between two processes. First, two attention models are proposed to exploit two types of correlation between the image and text for multi-modal embedding learning and to confuse the other process. Then the discriminator tries to distinguish the two types of multi-modal embeddings learned by the two attention models, in which the two attention models are reinforced mutually. Through adversarial learning, it is expected that both the two embeddings from the attention models can well exploit the two types of correlation, and thus they can deceive the discriminator that they are generated from the other attention-based model. By integrating the attention mechanism and adversarial learning, the learned multi-modal embeddings are more effective for image and text matching. Extensive experiments have been conducted on the benchmark datasets of Flickr30K and MSCOCO to demonstrate the superiority of the proposed approaches over the state-of-the-art methods on image-text retrieval.

Philosophy and Linguistics

This paper is an overview of the contemporary discussions in interdisciplinary studies of meaning of natural language expressions. The paper discusses the latest work published in two interdisciplinary journals «Linguistics and Philosophy» and «Natural Language Semantics» most relevant for philosophy of language. The paper focuses on two general topics: the semantics of singular terms (proper names, pronouns, demonstratives) and the semantics of belief-reports. The paper discusses the recent proposal about the interpretation of pronouns by [Stojnic et al., 2019] according to which such interpretation is strictly determined by the linguistic rules and does not depend on the context. According to this proposal, the referent of a pronoun is determined by a specific coherence relationship the sentence it occurs in has to the previous sentence in a discourse unless it is preceded by a pointing gesture. The paper discuses some issues with this proposal. Specifically, it discusses its difficulties with explaining the cases where the referent of a pronoun stands out in a context and, thus, no pointing gesture or previous discourse is required for the referent identification. It also discusses cases where a coherence relationship (such as Narration) between two sentences allows flexibility with respect to the referent identification and points at pragmatic factors that can be relevant in this respect. The paper also presents some other recent research on proper names, demonstratives and pronouns both linguistically and philosophically oriented.The paper also presents informally some work on belief-reports in linguistics and discussed its relevance for philosophy of language. It discusses the recent work on restrictions on the types of embeddings of propositional attitude verbs and the general approach that derives those restrictions from the predicted meanings of the relevant sentences, such as [Theiler et al., 2019]. The paper discussed the relevance of this issue to a more general question about the nature of a natural language and the relationship between language and logic.

An interval type-2 fuzzy TOPSIS using the extended vertex method for MAGDM

In this article, the technique of order preference by similarity to an ideal solution (TOPSIS) is modified to handle interval type-2 fuzzy sets (IT2FSs) using the extended vertex method for distance measure. While the existing TOPSIS techniques for IT2FSs depend on the defuzzification of the average decision matrix or the average weighted decision matrix from the very beginning, the proposed method maintains fuzziness in the preference technique up to the hilt to avoid any information distortion which might lead to false ranking. First, the vertex method for distance measure is extended to encompass IT2FSs. The extended vertex method is an efficient simple formula that requires few computations in contrast to other distance measures based on embedding type 1 fuzzy sets or α-cuts that need special algorithms and can be restrictive in applications that require high computations. Second, the fuzzy positive and negative ideal solutions are defined. Then, the relative degree of closeness to the ideal solutions is computed for each alternative using the extended vertex method. As the relative degree of closeness of an alternative increases, its preference increases. Therefore, the preference technique avoids the flaws of the existing techniques and the computations are reduced. Two illustrative examples are given and the results are compared with the results of the existing TOPSIS methods. In light of the results and comparisons, the role of the defined ideal solutions in ranking is clarified.

Recent advances in Swedish and Spanish medical entity recognition in clinical texts using deep neural approaches

BackgroundText mining and natural language processing of clinical text, such as notes from electronic health records, requires specific consideration of the specialized characteristics of these texts. Deep learning methods could potentially mitigate domain specific challenges such as limited access to in-domain tools and data sets.MethodsA bi-directional Long Short-Term Memory network is applied to clinical notes in Spanish and Swedish for the task of medical named entity recognition. Several types of embeddings, both generated from in-domain and out-of-domain text corpora, and a number of generation and combination strategies for embeddings have been evaluated in order to investigate different input representations and the influence of domain on the final results.ResultsFor Spanish, a micro averaged F1-score of 75.25 was obtained and for Swedish, the corresponding score was 76.04. The best results for both languages were achieved using embeddings generated from in-domain corpora extracted from electronic health records, but embeddings generated from related domains were also found to be beneficial.ConclusionsA recurrent neural network with in-domain embeddings improved the medical named entity recognition compared to shallow learning methods, showing this combination to be suitable for entity recognition in clinical text for both languages.

Leveraging More Fine-grained Representation to Reduce Instability within Word Embeddings

Suzi Park and Hyopil Shin. 2019. This paper investigates the notion of instability in word embeddings, primarily focused on the Korean language, by analyzing the effects of instability on several tasks using word level, character level, and sub-character level embeddings. Korean has a complex morphological system and is written at units smaller than that of a single character, making it optimal for comparing the three types of embeddings. We first carry out several intrinsic evaluations such as word similarity and analogy tests, as well as an extrinsic evaluation of the embeddings on the task of sentiment analysis. These results show that sub-character embeddings outperform the other two types, obtaining both higher and more consistent results, especially when the corpus size is large. Based on these findings, our hypothesis is that embeddings trained on smaller units and larger corpora would also be a better choice in terms of stability. To investigate this, we implemented two stability tests, looking into cosine similarity standard deviations and intersection of nearest neighbors among models. Our findings demonstrate that indeed, the smaller the embedding unit and the larger the corpus, the more stable the embedding type is. Importantly, our results illustrate the necessity of taking instability into consideration when using word embeddings within a given experiment.

SECNLP: A survey of embeddings in clinical natural language processing.

Distributed vector representations or embeddings map variable length text to dense fixed length vectors as well as capture prior knowledge which can transferred to downstream tasks. Even though embeddings have become de facto standard for text representation in deep learning based NLP tasks in both general and clinical domains, there is no survey paper which presents a detailed review of embeddings in Clinical Natural Language Processing. In this survey paper, we discuss various medical corpora and their characteristics, medical codes and present a brief overview as well as comparison of popular embeddings models. We classify clinical embeddings and discuss each embedding type in detail. We discuss various evaluation methods followed by possible solutions to various challenges in clinical embeddings. Finally, we conclude with some of the future directions which will advance research in clinical embeddings.

Reliable Classification of FAQs with Spelling Errors Using an Encoder-Decoder Neural Network in Korean

To resolve lexical disagreement problems between queries and frequently asked questions (FAQs), we propose a reliable sentence classification model based on an encoder-decoder neural network. The proposed model uses three types of word embeddings; fixed word embeddings for representing domain-independent meanings of words, fined-tuned word embeddings for representing domain-specific meanings of words, and character-level word embeddings for bridging lexical gaps caused by spelling errors. It also uses class embeddings to represent domain knowledge associated with each category. In the experiments with an FAQ dataset about online banking, the proposed embedding methods contributed to an improved performance of the sentence classification. In addition, the proposed model showed better performance (with an accuracy of 0.810 in the classification of 411 categories) than that of the comparison model.

Rectangular W algebras and superalgebras and their representations

We study W-algebras obtained by quantum Hamiltonian reduction of $sl(Mn)$ associated to the $sl(2)$ embedding of rectangular type. The algebra can be realized as the asymptotic symmetry of higher spin gravity with $M \times M$ matrix valued fields. In our previous work, we examined the basic properties of the W-algebra and claimed that the algebra can be realized as the symmetry of Grassmannian-like coset even with finite central charge based on a proposal of holography. In this paper, we extend the analysis in the following ways. First, we compute the operator product expansions among low spin generators removing the restriction of $n = 2$. Second, we investigate the degenerate representations in several ways, and see the relations to the coset spectrum and the conical defect geometry of the higher spin gravity. For these analyses, we mainly set $M=n=2$. Finally, we extend the previous analysis by introducing $\mathcal{N}=2$ supersymmetry.

Word Embeddings for IoT Based on Device Activity Footprints

With the expansion of IoT ecosystem, there is an explosion of the number of devices and sensors and the data generated by these devices. However, the tools available to analyze such data are limited. Word embeddings, widely used in the natural language processing (NLP) domain, provides a way to get similar words to the current word. In this paper, we extend the theory of word embeddings to the area of IoT devices, proposing a method to generate the word embeddings for IoT devices and sensors in a smart home based on their activity. We model IoT devices as vectors using a concept like Word2Vec and App2Vec, where the time between the device firings is also taken into account. These computed word embeddings can be used for a variety of use cases, such as to find similar devices in an IoT device store, or as a signature of each type of IoT device. We show results of a feasibility study on the CASAS dataset and a private real-world dataset of IoT device activity logs, using our method to identify the patterns in embeddings of various types of IoT devices in a household. We get a probability of more than 0.65 for similar types of devices clustering together, independent of session gap value and embedding vector size for the CASAS dataset. We also get a prob-ability of 0.4 on the private dataset, independent of session gap value and embedding vector size.