Vector Space Embedding Research Articles

Extracting Features from Text Flows based on Semantic Similarity for Text Classification: an Approach Inspired by Audio Analysis

Text classification is a mainly investigated challenge in Natural Language Processing (NLP) research. The higher performance of a classification model depends on a representation that can extract valuable information about the texts. Aiming not to lose crucial local text information, a way to represent texts is through flows, sequences of information collected from texts. This paper proposes an approach that combines various techniques to represent texts: the representation by flows, the benefit of the word embeddings text representation associated with lexicon information via semantic similarity distances, and the extraction of features inspired by well-established audio analysis features.In order to perform text classification, this approach splits the text into sentences and calculates a semantic similarity metric to a lexicon on an embedding vector space. The sequence of semantic similarity metrics composes the text flow. Then, the method performs the extraction of twenty-five features inspired by audio analysis (named Audio-Like Features). The features adaptation from audio analysis comes from a similitude between a text flow and a digital signal, in addition to the existing relationship between text, speech, and audio. We evaluated the method in three NLP classification tasks: Fake News Detection in English, Fake News Detection in Portuguese, and Newspaper Columns versus News Classification. The approach efficacy is compared to baselines that embed semantics in text representation: the Paragraph Vector and the BERT. The objective of the experiments was to investigate if the proposed approach could compete with the baselines methods improve their efficacy when associated with them. The experimental evaluation demonstrates that the association between the proposed and the baseline methods can enhance the baseline classification efficacy in all three scenarios. In the Fake News Detection in Portuguese task, our approach surpassed the baselines and obtained the best effectiveness (PR-AUC = 0.98).

Quantifying advances from basic research to applied research in material science

Exploring the connection between basic science and its practical applications is critical to consider the social justification of the substantial governmental investments in science. However, there's a limited understanding of global research patterns and the dynamics of researcher collaboration from a basic-applied perspective because such studies have been mainly focused only on the biomedical field. Here, the main goal is to propose an indicator to quantify the degree of basic-applied research in academic papers. Using the indicator, we uncover how material science has advanced from basic to applied research, based on the international trends of the indicators and the affiliations of the scientists involved. We develop a methodology that indexes levels of advancement from basic research to applied research based on large-scale text data. The continuous scores assigned to each paper are derived from a vector space embedding technical terms from a broad network data. These scores align with experts' views in material science. This methodology enables us to monitor international trends that China has significantly advanced into applied research, as well as Chinese applied scientists increasingly associating with their domestic institutions. As science and technology implication, our methodology extends the boundary of assessing scientific research on its proximity to real-world applications and provides a tangible measure for funding agencies managing to fund or design research environments.

Evaluation of word embedding models used for diachronic semantic change analysis

In the last decade, the quantitative analysis of diachronic changes in language and lexical semantic changes have become the subject of active research. A significant role was played by the development of new effective techniques of word embedding. This direction has been effectively demonstrated in a number of studies. Some of them have focused on the analysis of the optimal type of word2vec models, hyperparameters for training, and evaluation techniques. In this research, we used Corpus of Historical American English (COHA). The paper demonstrates the results of multiple training runs and the comparison of word2vec models with different variations of hyperparameters used for lexical semantic change detection. In addition to traditional word similarities and analogical reasoning tests, we used testing on an extended set of synonyms. We have evaluated word2vec models on the set of more than 100,000 English synsets that were randomly selected from the WordNet database. We have shown that changing the word2vec model parameters (such as a dimension of word embedding, a size of context window, a type of model, a word discard rate etc.) can significantly impact on the resulting word embedding vector space and the detected lexical semantic changes. Additionally, the results strongly depended on properties of the corpus, such as word frequency distribution.

Hyperbolic Actions and 2nd Bounded Cohomology of Subgroups of 𝖮𝗎𝗍(𝖥_{𝗇})

In this two part work we prove that for every finitely generated subgroup Γ > O u t ( F n ) \Gamma >{\mathsf {Out}}(F_n) , either Γ \Gamma is virtually abelian or H b 2 ( Γ ; R ) H^2_b(\Gamma ;{\mathbb {R}}) contains a vector space embedding of ℓ 1 \ell ^1 . The method uses actions on hyperbolic spaces. In Part I we focus on the case of infinite lamination subgroups Γ \Gamma —those for which the set of all attracting laminations of all elements of Γ \Gamma is an infinite set—using actions on free splitting complexes of free groups. In Part II we focus on finite lamination subgroups Γ \Gamma and on the construction of useful new hyperbolic actions of those subgroups.

SpeckleNN: a unified embedding for real-time speckle pattern classification in X-ray single-particle imaging with limited labeled examples.

With X-ray free-electron lasers (XFELs), it is possible to determine the three-dimensional structure of noncrystalline nanoscale particles using X-ray single-particle imaging (SPI) techniques at room temperature. Classifying SPI scattering patterns, or `speckles', to extract single-hits that are needed for real-time vetoing and three-dimensional reconstruction poses a challenge for high-data-rate facilities like the European XFEL and LCLS-II-HE. Here, we introduce SpeckleNN, a unified embedding model for real-time speckle pattern classification with limited labeled examples that can scale linearly with dataset size. Trained with twin neural networks, SpeckleNN maps speckle patterns to a unified embedding vector space, where similarity is measured by Euclidean distance. We highlight its few-shot classification capability on new never-seen samples and its robust performance despite having only tens of labels per classification category even in the presence of substantial missing detector areas. Without the need for excessive manual labeling or even a full detector image, our classification method offers a great solution for real-time high-throughput SPI experiments.

Entity Alignment Method Based on Joint Learning of Entity and Attribute Representations

Entity alignment helps discover and link entities from different knowledge graphs (KGs) that refer to the same real-world entity, making it a critical technique for KG fusion. Most entity alignment methods are based on knowledge representation learning, which uses a mapping function to project entities from different KGs into a unified vector space and align them based on calculated similarities. However, this process requires sufficient pre-aligned entity pairs. To address this problem, this study proposes an entity alignment method based on joint learning of entity and attribute representations. Structural embeddings are learned using the triples modeling method based on TransE and PTransE and extracted from the embedding vector space utilizing semantic information from direct and multi-step relation paths. Simultaneously, attribute character embeddings are learned using the N-gram-based compositional function to encode a character sequence for the attribute values, followed by TransE to model attribute triples in the embedding vector space to obtain attribute character embedding vectors. By learning the structural and attribute character embeddings simultaneously, the structural embeddings of entities from different KGs can be transferred into a unified vector space. Lastly, the similarities in the structural embedding of different entities were calculated to perform entity alignment. The experimental results showed that the proposed method performed well on the DBP15K and DWK100K datasets, and it outperformed currently available entity alignment methods by 16.8, 27.5, and 24.0% in precision, recall, and F1 measure, respectively.

322 Distributed Vector Representations of Neurosurgical Admission Notes Predict Discharge Disposition

INTRODUCTION: Natural language processing (NLP) aims to extract information from unstructured language input for use in machine learning, without the need for laborious manual review of text to retrieve data of potential interest. An NLP-based model utilizing term frequency-inverse document frequency (TF-IDF) vectoral representations of pre-operative clinical notes has previously been shown to predict discharge disposition following meningioma resection. The TF-IDF vectorization scheme is based on word counts across documents but does not account for word order or semantic information. An alternative algorithm, paragraph vectorization, utilizes an unsupervised neural network to learn document vector representations. METHODS: We analyzed 7,122 admission notes. We trained a neural network to derive paragraph vector representations of notes and trained a classifier to predict home vs non-home discharge disposition based on these representations. Out-of-sample performance was assessed on a held-out testing set. We repeated the analysis using TF-IDF document representations to compare the two approaches. Performance was quantified using area under the receiver operating characteristic curve (ROC-AUC). RESULTS: Model performance increased with vector dimensionality. A 200-dimensional distributed document vectorization was highly predictive of nonhome discharge with an ROC-AUC of 0.82 ± 0.01 (mean ± standard error over 5-fold nested cross-validation). When the dimensionality of the embedding vector space was relatively low (e.g. <100), learned paragraph vectors predicted discharge disposition better than TF-IDF vectors. CONCLUSIONS: Natural language processing of admission notes via neural network-learned distributed vectoral representations is a viable strategy of information extraction from neurosurgical clinical documentation. This embedding scheme overcomes some of the limitations of TF-IDF and encodes more information in lower-dimensional embedding vector spaces. It holds promise as a method of using free text to train machine learning algorithms for prediction of clinically relevant outcomes.

Neuromorphic Similarity Measurement of Tactile Stimuli in Human–Machine Interface

The interpretation of tactile stimuli empowers humans to identify substances, distinguish materials, and engage in tactile communication. For stimulus design in human-computer interaction, objective similarity measures improve efficiency and save costs. Inspired by the fact that biological systems are robust in recognizing multimedia stimuli, we propose a neuromorphic method for similarity measurement. The method is divided into two steps. First, tactile information is translated into biological representations by mimicking a low-threshold mechanoreceptor through a physiological neuronal model. Then, three measures are nominated to assess the similarity of neural spike trains from the following perspectives: interval spike counting, temporal matching, and vector space embedding. Regression analysis showed that the linearity of these measures was significant, indicating that the filtering ability of the physiological neuron model is robust. One of the measures is selected for comparison with the signal-to-noise ratio, structural similarity, and hybrid metric. The results suggest that the correlation between the predictions of our method and the subjective evaluation is stable, above 0.9 for each experimental stimulus. We achieve a mutual interpretation between quantitative measures of vibrotactile similarity and subjective cognitive outcomes. Furthermore, the feasibility of this method in material classification has been substantiated through an exploratory experiment.

Neural Embedding Allocation: Distributed Representations of Topic Models

Abstract We propose a method that uses neural embeddings to improve the performance of any given LDA-style topic model. Our method, called neural embedding allocation (NEA), deconstructs topic models (LDA or otherwise) into interpretable vector-space embeddings of words, topics, documents, authors, and so on, by learning neural embeddings to mimic the topic model. We demonstrate that NEA improves coherence scores of the original topic model by smoothing out the noisy topics when the number of topics is large. Furthermore, we show NEA’s effectiveness and generality in deconstructing and smoothing LDA, author-topic models, and the recent mixed membership skip-gram topic model and achieve better performance with the embeddings compared to several state-of-the-art models.

A Novel Cross-Network Embedding for Anchor Link Prediction with Social Adversarial Attacks

Anchor link prediction across social networks plays an important role in multiple social network analysis. Traditional methods rely heavily on user privacy information or high-quality network topology information. These methods are not suitable for multiple social networks analysis in real-life. Deep learning methods based on graph embedding are restricted by the impact of the active privacy protection policy of users on the graph structure. In this paper, we propose a novel method which neutralizes the impact of users’ evasion strategies. First, graph embedding with conditional estimation analysis is used to obtain a robust embedding vector space. Secondly, cross-network features space for supervised learning is constructed via the constraints of cross-network feature collisions. The combination of robustness enhancement and cross-network feature collisions constraints eliminate the impact of evasion strategies. Extensive experiments on large-scale real-life social networks demonstrate that the proposed method significantly outperforms the state-of-the-art methods in terms of precision, adaptability, and robustness for the scenarios with evasion strategies.

SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

Heuristic Attention Representation Learning for Self-Supervised Pretraining.

Recently, self-supervised learning methods have been shown to be very powerful and efficient for yielding robust representation learning by maximizing the similarity across different augmented views in embedding vector space. However, the main challenge is generating different views with random cropping; the semantic feature might exist differently across different views leading to inappropriately maximizing similarity objective. We tackle this problem by introducing Heuristic Attention Representation Learning (HARL). This self-supervised framework relies on the joint embedding architecture in which the two neural networks are trained to produce similar embedding for different augmented views of the same image. HARL framework adopts prior visual object-level attention by generating a heuristic mask proposal for each training image and maximizes the abstract object-level embedding on vector space instead of whole image representation from previous works. As a result, HARL extracts the quality semantic representation from each training sample and outperforms existing self-supervised baselines on several downstream tasks. In addition, we provide efficient techniques based on conventional computer vision and deep learning methods for generating heuristic mask proposals on natural image datasets. Our HARL achieves +1.3% advancement in the ImageNet semi-supervised learning benchmark and +0.9% improvement in AP50 of the COCO object detection task over the previous state-of-the-art method BYOL. Our code implementation is available for both TensorFlow and PyTorch frameworks.

Word embeddings are biased. But whose bias are they reflecting?

From Curriculum Vitae parsing to web search and recommendation systems, Word2Vec and other word embedding techniques have an increasing presence in everyday interactions in human society. Biases, such as gender bias, have been thoroughly researched and evidenced to be present in word embeddings. Most of the research focuses on discovering and mitigating gender bias within the frames of the vector space itself. Nevertheless, whose bias is reflected in word embeddings has not yet been investigated. Besides discovering and mitigating gender bias, it is also important to examine whether a feminine or a masculine-centric view is represented in the biases of word embeddings. This way, we will not only gain more insight into the origins of the before mentioned biases, but also present a novel approach to investigating biases in Natural Language Processing systems. Based on previous research in the social sciences and gender studies, we hypothesize that masculine-centric, otherwise known as androcentric, biases are dominant in word embeddings. To test this hypothesis we used the largest English word association test data set publicly available. We compare the distance of the responses of male and female participants to cue words in a word embedding vector space. We found that the word embedding is biased towards a masculine-centric viewpoint, predominantly reflecting the worldviews of the male participants in the word association test data set. Therefore, by conducting this research, we aimed to unravel another layer of bias to be considered when examining fairness in algorithms.

Stock portfolio selection balancing variance and tail risk via stock vector representation acquired from price data and texts

Recent works on portfolio selection report ways to incorporate textual data in addition to price movements. Price, texts, and events as what lies underneath take heterogeneous data form and therefore have been processed without any consistent mathematical formulation.In this article, we propose to generalize portfolio selection by representing all related objects (stocks, news, events) in an embedding vector space, that we call a NEws-STock space with Event Distribution (NESTED). A NESTED forms an inner product vector space (Hilbert space), in which texts and stocks are represented as vectors (embeddings), acquired through a distribution of events. In this article, we first theoretically reformulate Markowitz’s portfolio optimization problem on NESTED. We show how our new formulation has the potential to better incorporate the tail risk, which is represented better in textual data.One typical method to acquire such embeddings is via neural computing. Our experimental results, obtained by using it on 24 news-price datasets across three markets, showed that the Pareto’s exponent in the negative tail of the generated portfolios increased in all markets, which is evidence that the stock embeddings captured the tail risks. Our method showed a large improvement balancing between the tail risk and non-tail risk, up to 45.5% larger gain and 59.4% larger Information ratio.

Improving medical term embeddings using UMLS Metathesaurus

BackgroundHealth providers create Electronic Health Records (EHRs) to describe the conditions and procedures used to treat their patients. Medical notes entered by medical staff in the form of free text are a particularly insightful component of EHRs. There is a great interest in applying machine learning tools on medical notes in numerous medical informatics applications. Learning vector representations, or embeddings, of terms in the notes, is an important pre-processing step in such applications. However, learning good embeddings is challenging because medical notes are rich in specialized terminology, and the number of available EHRs in practical applications is often very small.MethodsIn this paper, we propose a novel algorithm to learn embeddings of medical terms from a limited set of medical notes. The algorithm, called definition2vec, exploits external information in the form of medical term definitions. It is an extension of a skip-gram algorithm that incorporates textual definitions of medical terms provided by the Unified Medical Language System (UMLS) Metathesaurus.ResultsTo evaluate the proposed approach, we used a publicly available Medical Information Mart for Intensive Care (MIMIC-III) EHR data set. We performed quantitative and qualitative experiments to measure the usefulness of the learned embeddings. The experimental results show that definition2vec keeps the semantically similar medical terms together in the embedding vector space even when they are rare or unobserved in the corpus. We also demonstrate that learned vector embeddings are helpful in downstream medical informatics applications.ConclusionThis paper shows that medical term definitions can be helpful when learning embeddings of rare or previously unseen medical terms from a small corpus of specialized documents such as medical notes.

Reviewer Recommendations Using Document Vector Embeddings and a Publisher Database: Implementation and Evaluation

We develop and evaluate an automated data-driven framework for providing reviewer recommendations for submitted manuscripts. Given inputs comprising a set of manuscripts for review and a listing of a pool of prospective reviewers, our system uses a publisher database to extract papers authored by the reviewers from which a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Paragraph Vector</i> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">doc2vec</i> ) neural network model is learned and used to obtain vector space embeddings of documents. Similarities between embeddings of an individual reviewer’s papers and a manuscript are then used to compute manuscript-reviewer match scores and to generate a ranked list of recommended reviewers for each manuscript. Our mainline proposed system uses full text versions of the reviewers’ papers, which we demonstrate performs significantly better than models developed based on abstracts alone, which has been the predominant paradigm in prior work. Direct retrieval of reviewer’s manuscripts from a publisher database reduces reviewer burden, ensures up-to-date data, and eliminates the potential for misuse through data manipulation. We also propose a useful evaluation methodology that addresses hyperparameter selection and enables indirect comparisons with alternative approaches and on prior datasets. Finally, the work also contributes a large scale retrospective reviewer matching dataset and evaluation that we hope will be useful for further research in this field. Our system is quite effective; for the mainline approach, expert judges rated 38% of the recommendations as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Very Relevant</i> ; 33% as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Relevant</i> ; 24% as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Slightly Relevant</i> ; and only 5% as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Irrelevant</i> .

Preliminary Analysis of the Risk Factor Identification Embedding Model for Cardiovascular Disease.

Cardiovascular Disease (CVD) is responsible for a large part of healthcare costs every year, but susceptibility to it is affected by complex biological and physiological variables including patients' genetics and lifestyles. There has not been much work to develop a framework that incorporates these important and clinically relevant risk factors into a comprehensive model for CVD research. Moreover, the data labeling required to do so, such as annotating gene functions, is an extremely challenging, tedious, and time-consuming process. In this work, our goal was to develop and validate a risk factor embedding model, which incorporates genotype, phenotype without pre-labeled information to identify various risk factors of CVD. We hypothesize that (1) the knowledge background that does not require data labeling could be gathered from published abstract data, (2) the phenotype, genotype risk factors could be represented in an embedding vector space. We collected 1,363,682 published abstracts from PubMed using the keyword "heart" and 19,264 human gene names, then trained our model using the collected abstracts. We evaluated our CVD risk factor identification model using both intrinsic and extrinsic evaluations: for the intrinsic evaluation, we examined whether or not the captured top-10 words and genes have references related to the input query "myocardial infarction", as one of CVDs, and our model correctly identified them. For the extrinsic evaluation, we used our model to the dimensionality reduction task for classifications, and our method outperformed other popular methods. These results show the feasibility of our approach for disease-associated risk factors of CVD which incorporates genotype, phenotype.Clinical Relevance-Our model provides a comprehensive tool to incorporate various risk factors without any a priori data labeling knowledge for CVD. Our approach shows a potential to provide discovered knowledge that contributes to better understanding and treatment of CVD.

Distance-Preserving Vector Space Embedding for Consensus Learning

Learning a prototype from a set of given objects is a core problem in machine learning and pattern recognition. A commonly used approach for consensus learning is to formulate it as an optimization problem in terms of generalized median computation. Recently, a prototype-embedding approach has been proposed to transform the objects into a vector space, compute the geometric median, and then inversely transform back into the original space. This vector space embedding approach has been successfully applied in several domains, where the generalized median problem has inherent high-computational complexity (typically NP-hard) and thus approximate solutions are required. Generally, it can be expected that the embedding should be done in a distance-preserving manner. However, the previous work based on the prototype-embedding approach did not take this embedding aspect into account. In this paper, we discuss the drawbacks of the current prototype-embedding approach and present an extensive empirical study that provides strong evidence of significantly improved quality of generalized median computation using distance-preserving embedding (DPE) methods. We also give concrete suggestions about suitable DPE methods. Moreover, we show that this framework can be used to effectively compute other consensus objects like the closest string. Finally, a MATLAB toolbox resulting from this paper is made publically available in order to encourage other researchers to explore the embedding-based consensus computation.

Robust Visual Relationship Detection towards Sparse Images in Internet‐of‐Things

Visual relationship can capture essential information for images, like the interactions between pairs of objects. Such relationships have become one prominent component of knowledge within sparse image data collected by multimedia sensing devices. Both the latent information and potential privacy can be included in the relationships. However, due to the high combinatorial complexity in modeling all potential relation triplets, previous studies on visual relationship detection have used the mixed visual and semantic features separately for each object, which is incapable for sparse data in IoT systems. Therefore, this paper proposes a new deep learning model for visual relationship detection, which is a novel attempt for cooperating computational intelligence (CI) methods with IoTs. The model imports the knowledge graph and adopts features for both entities and connections among them as extra information. It maps the visual features extracted from images into the knowledge‐based embedding vector space, so as to benefit from information in the background knowledge domain and alleviate the impacts of data sparsity. This is the first time that visual features are projected and combined with prior knowledge for visual relationship detection. Moreover, the complexity of the network is reduced by avoiding the learning of redundant features from images. Finally, we show the superiority of our model by evaluating on two datasets.

Multimodal Few-Shot Learning for Gait Recognition

A person’s gait is a behavioral trait that is uniquely associated with each individual and can be used to recognize the person. As information about the human gait can be captured by wearable devices, a few studies have led to the proposal of methods to process gait information for identification purposes. Despite recent advances in gait recognition, an open set gait recognition problem presents challenges to current approaches. To address the open set gait recognition problem, a system should be able to deal with unseen subjects who have not included in the training dataset. In this paper, we propose a system that learns a mapping from a multimodal time series collected using insole to a latent (embedding vector) space to address the open set gait recognition problem. The distance between two embedding vectors in the latent space corresponds to the similarity between two multimodal time series. Using the characteristics of the human gait pattern, multimodal time series are sliced into unit steps. The system maps unit steps to embedding vectors using an ensemble consisting of a convolutional neural network and a recurrent neural network. To recognize each individual, the system learns a decision function using a one-class support vector machine from a few embedding vectors of the person in the latent space, then the system determines whether an unknown unit step is recognized as belonging to a known individual. Our experiments demonstrate that the proposed framework recognizes individuals with high accuracy regardless they have been registered or not. If we could have an environment in which all people would be wearing the insole, the framework would be used for user verification widely.