Word Mover Research Articles

Synwmd: Syntax-Aware Word Mover's Distance for Sentence Similarity Evaluation

Word Mover's Distance (WMD) computes the distance between words and models text similarity with the moving cost between words in two text sequences. Yet, it does not offer good performance in sentence similarity evaluation since it does not incorporate word importance and fails to take inherent contextual and structural information in a sentence into account. An improved WMD method using the syntactic parse tree, called Syntax-aware Word Mover's Distance (SynWMD), is proposed to address these two shortcomings in this work. First, a weighted graph is built upon the word co-occurrence statistics extracted from the syntactic parse trees of sentences. The importance of each word is inferred from graph connectivities. Second, the local syntactic parsing structure of words is considered in computing the distance between words. To demonstrate the effectiveness of the proposed SynWMD, we conduct experiments on 6 textual semantic similarity (STS) datasets and 4 sentence classification datasets. Experimental results show that SynWMD achieves state-of-the-art performance on STS tasks. It also outperforms other WMD-based methods on sentence classification tasks.

Evaluation of Descriptive Exam Answer Scripts using Word Mover’s Distance

The knowledge and competency assessment have paramount significance in the education system. Recent scenario of COVID-19 witnessed the need of migrating from traditional education system to a modern online learning environment. Currently in the online assessment process, descriptive exam answer scripts evaluation is one of the tedious tasks to the teachers. The knowledge assessment may sometimes lead to biasing based on the mood of the evaluator and other circumstancing parameters. In general, though the evaluation process is well defined, still when two evaluators evaluate the same scripts, there are very less chances to award the same marks. The proposed model aims to address such real time issues and outer performs of the evaluation of descriptive answer scripts by using text semantic similarity measure. The proposed model works based on the word mover’s distance, whose purpose is to measure the semantic similarity among the actual answer and the answer given by the students. In this work, the data set is generated from the descriptive on-line examination platform. The data set contains student’s answers, which can pre-process initially and measure the semantic similarity among key answer and student’s answers. The given automatic evaluation procedure, could guarantee the impartiality and concealment of the evaluation.

Multi-Ideology ISIS/Jihadist White Supremacist (MIWS) Dataset for Multi-Class Extremism Text Classification

Social media platforms are a popular choice for extremist organizations to disseminate their perceptions, beliefs, and ideologies. This information is generally based on selective reporting and is subjective in content. However, the radical presentation of this disinformation and its outreach on social media leads to an increased number of susceptible audiences. Hence, detection of extremist text on social media platforms is a significant area of research. The unavailability of extremism text datasets is a challenge in online extremism research. The lack of emphasis on classifying extremism text into propaganda, radicalization, and recruitment classes is a challenge. The lack of data validation methods also challenges the accuracy of extremism detection. This research addresses these challenges and presents a seed dataset with a multi-ideology and multi-class extremism text dataset. This research presents the construction of a multi-ideology ISIS/Jihadist White supremacist (MIWS) dataset with recent tweets collected from Twitter. The presented dataset can be employed effectively and importantly to classify extremist text into popular types like propaganda, radicalization, and recruitment. Additionally, the seed dataset is statistically validated with a coherence score of Latent Dirichlet Allocation (LDA) and word mover’s distance using a pretrained Google News vector. The dataset shows effectiveness in its construction with good coherence scores within a topic and appropriate distance measures between topics. This dataset is the first publicly accessible multi-ideology, multi-class extremism text dataset to reinforce research on extremism text detection on social media platforms.

Machine Learning for Creativity: Using Similarity Networks to Design Better Crowdfunding Projects

A fundamental tension exists in creativity between novelty and similarity. This research exploits this tension to help creators craft successful projects in crowdfunding. To do so, the authors apply the concept of combinatorial creativity, analyzing each new project in connection to prior similar projects. By using machine learning techniques (Word2vec and Word Mover’s Distance), they measure the degrees of similarity between crowdfunding projects on Kickstarter. They analyze how this similarity pattern relates to a project’s funding performance and find that (1) the prior level of success of similar projects strongly predicts a new project’s funding performance, (2) the funding performance increases with a balance between being novel and imitative, (3) the optimal funding goal is close to the funds raised by prior similar projects, and (4) the funding performance increases with a balance between atypical and conventional imitation. The authors use these findings to generate actionable recommendations for project creators and crowdfunding platforms.

Semrank

Popular real-world events often create huge traffic on Twitter including real-time updates of important moments, personal comments, and so on while the event is happening. Most of the users are interested to read the important tweets that possibly include important moments of that event. However, extracting the relevant tweets of any event is a challenging task due to the endless stream of noisy tweets and vocabulary variation problem of social media content. To handle these challenges, the authors introduce a new approach for computing the relative tweet importance based on the concept of the Pagerank algorithm where adjacency matrix of the graph representation of tweets contains semantic similarity matrix based on the word mover's distance measure utilizing Word2Vec word embedding model. The results show that top-ranked tweets generated by the proposed approach are more concise and news-worthy than baseline approaches.

DETERMINING THE SEMANTIC PROXIMITY OF NEWS MESSAGES BASED ON TITLES ANALYSIS

The paper is devoted to the peculiarities of using data from unstructured sources, such as social networks, online media, etc. for the development of management decisions. The issue of analyzing such sources in the process of developing and making managerial decisions in the socio-economic sphere is considered. The difficulties of processing unstructured data are described, such as the impossibility of fully automatic evaluation of the semantics of data, the presence of a large amount of duplicate information. Various approaches to determining ratings and various metrics of sources and messages are described. The problem of identifying duplicate messages is considered using the example of online media both by full texts and by titles. Under the duplicate of a news message in this context refers a repetition of a significant amount of information from it in another article. If the text of a news message (article) is not repeated verbatim, it is impossible to determine the degree of duplication of the meaning of the article without involving human expertise. There are various metrics for assessing the similarity (semantic proximity) of textual information that can help in this matter. Some of them are described in the article. An adaptation of the Word Mover Distance method for the Russian language is proposed and the Word2Vec model is trained for its use. A hybrid approach to identifying and eliminating duplicate messages as part of the preprocessing of unstructured data in the context of managerial decision-making is proposed. According to the results of the experiments, depending on the chosen method, it was possible to automatically determine based on the publication time and the analysis of titles from 43 to 74 % duplicates.

Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric

The rapid development of such natural language processing tasks as style transfer, paraphrase, and machine translation often calls for the use of semantic similarity metrics. In recent years a lot of methods to measure the semantic similarity of two short texts were developed. This paper provides a comprehensive analysis for more than a dozen of such methods. Using a new dataset of fourteen thousand sentence pairs human-labeled according to their semantic similarity, we demonstrate that none of the metrics widely used in the literature is close enough to human judgment in these tasks. A number of recently proposed metrics provide comparable results, yet Word Mover Distance is shown to be the most reasonable solution to measure semantic similarity in reformulated texts at the moment.

Topic extraction from extremely short texts with variational manifold regularization

With the emerging of massive short texts, e.g., social media posts and question titles from Q&A systems, discovering valuable information from them is increasingly significant for many real-world applications of content analysis. The family of topic modeling can effectively explore the hidden structures of documents through the assumptions of latent topics. However, due to the sparseness of short texts, the existing topic models, e.g., latent Dirichlet allocation, lose effectiveness on them. To this end, an effective solution, namely Dirichlet multinomial mixture (DMM), supposing that each short text is only associated with a single topic, indirectly enriches document-level word co-occurrences. However, DMM is sensitive to noisy words, where it often learns inaccurate topic representations at the document level. To address this problem, we extend DMM to a novel Laplacian Dirichlet Multinomial Mixture (LapDMM) topic model for short texts. The basic idea of LapDMM is to preserve local neighborhood structures of short texts, enabling to spread topical signals among neighboring documents, so as to modify the inaccurate topic representations. This is achieved by incorporating the variational manifold regularization into the variational objective of DMM, constraining the close short texts with similar variational topic representations. To find nearest neighbors of short texts, before model inference, we construct an offline document graph, where the distances of short texts can be computed by the word mover’s distance. We further develop an online version of LapDMM, namely Online LapDMM, to achieve inference speedup on massive short texts. Carrying this implications, we exploit the spirit of stochastic optimization with mini-batches and an up-to-date document graph that can efficiently find approximate nearest neighbors instead. To evaluate our models, we compare against the state-of-the-art short text topic models on several traditional tasks, i.e., topic quality, document clustering and classification. The empirical results demonstrate that our models achieve very significant performance gains over the baseline models.

Extractive multi-document text summarization using dolphin swarm optimization approach

Nowadays, extracting the desired information from internet source is a challenging task because of a large amount of information available on the internet. So, we propose a new extractive based approach for multi-document text summarization to extract useful information from multi-document. Initially, the redundant contents in the document create a single text file from the multiple text file document. The content coverage and non-redundancy features are achieved by Word Mover Distance (WMD) and Modified Normalized Google Distance (M-NGD) (WM) Hybrid Weight Method based similarity approaches. For feature weight optimization, we use the Dolphin swarm optimization (DSO) which is a metaheuristic approach. The proposed approach is tested under python with multiling 2013 dataset and the performances have been evaluated with ROUGE and AutoSummENG metrics. The investigational outcomes show that the proposed technique works well and very much effective for multi-document text summarization.

Reinforced Abstractive Text Summarization With Semantic Added Reward

Text summarization is an important task in natural language processing (NLP). Neural summary models summarize information by understanding and rewriting documents through the encoder-decoder structure. Recent studies have sought to overcome the bias that cross-entropy-based learning methods can have through reinforcement learning (RL)-based learning methods or the problem of failing to learn optimized for metrics. However, the ROUGE metric with only <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> -gram matching is not a perfect solution. The purpose of this study is to improve the quality of the summary statement by proposing a reward function used in text summarization based on RL. We propose ROUGE-SIM and ROUGE-WMD, modified functions of the ROUGE function. ROUGE-SIM enables meaningfully similar words, in contrast to ROUGE-L. ROUGE-WMD is a function adding semantic similarity to ROUGE-L. The semantic similarity between articles and summary text was computed using Word Mover’s Distance (WMD) methodology. Our model with two proposed reward functions demonstrated superior performance on ROUGE-1, ROUGE-2, and ROUGE_L than on ROUGE-L as a reward function. Our two models, ROUGE-SIM and ROUGE-WMD, scored 0.418 and 0.406 for ROUGE-L, respectively, for the Gigaword dataset. The two reward functions outperformed ROUGE-L even in the abstractiveness and grammatical aspects.

Toward Tweet-Mining Framework for Extracting Terrorist Attack-Related Information and Reporting

The widespread popularity of social networking is leading to the adoption of Twitter as an information dissemination tool. Existing research has shown that information dissemination over Twitter has a much broader reach than traditional media and can be used for effective post-incident measures. People use informal language on Twitter, including acronyms, misspelled words, synonyms, transliteration, and ambiguous terms. This makes incident-related information extraction a non-trivial task. However, this information can be valuable for public safety organizations that need to respond in an emergency. This paper proposes an early event-related information extraction and reporting framework that monitors Twitter streams synthesizes event-specific information, e.g., a terrorist attack, and alerts law enforcement, emergency services, and media outlets. Specifically, the proposed framework, Tweet-to-Act (T2A), employs word embedding to transform tweets into a vector space model and then utilizes the Word Mover’s Distance (WMD) to cluster tweets for the identification of incidents. To extract reliable and valuable information from a large dataset of short and informal tweets, the proposed framework employs sequence labeling with bidirectional Long Short-Term Memory based Recurrent Neural Networks ( bLSTM-RNN ). Extensive experimental results suggest that our proposed framework, T2A, outperforms other state-of-the-art methods that use vector space modeling and distance calculation techniques, e.g., Euclidean and Cosine distance. T2A achieves an accuracy of 96% and an F1-score of 86.2% on real-life datasets.

Design and Implementation of the Traditional Chinese Medicine Constitution System Based on the Diagnosis of Tongue and Consultation

The standardization and digitalization of nine constitutions of traditional Chinese medicine (TCM) have promoted the development of TCM automation. The keys to constitution identification are tongue diagnosis and consultation diagnosis. In this paper, a tongue detection method based on the combination of the histogram of oriented gradients (HOG) and the support vector machine (SVM) is proposed. To separate the tongue body and tongue coating, a k-means segmentation method based on the Lab color space is proposed. Based on the clustering analysis of the difference between the color components of the tongue body and tongue coating in the Lab color space, the separation is realized. Thus, the relationship between the tongue image and constitution can be analyzed. The method for consultation diagnosis is divided into a questionnaire scale and a question answering system. The questionnaire includes 29 quantified items with corresponding weights and answer scores, so the final score of each type of constitution is calculated. The final constitution type is determined on the basis of the scores of both diagnoses. In addition, the key of question answering system is text similarity calculation. First, a sentence similarity calculation method combining the n-gram language model and word2vec is proposed. Then, a multifeature fusion sentence similarity calculation method combining the Word Mover’s Distance (WMD) and editing distance is proposed. Finally, the experimental results show the effectiveness of the proposed method, and a constitution identification application is developed based on the content of this paper.

Enfoque Combinado de Word2Vec y 2-grams para la Recuperación de Avisos Clasificados Inmobiliarios Semánticamente Relacionados

La publicación de avisos clasificados de inmuebles se ha convertido en el medio de publicidad preferido tanto para particulares como empresas inmobiliarias. Esto ha provocado un crecimiento importante en la cantidad de avisos, tornando difícil la búsqueda un inmueble adecuado, mucho más si la búsqueda es en una gran ciudad. Este trabajo propone un enfoque basado en técnicas de minería de textos y procesamiento del lenguaje natural para la recuperación de avisos clasificados semánticamente relacionados. Para tal propósito se recolectaron los avisos publicados por el sitio web lavoz.com.ar, mediante un scraper. El título y la descripción de estos avisos fueron empleados para conformar un corpus textual modelado mediante Word2Vec, evaluando la similitud por medio de Word Mover’s Distance. El empleo de 2-grams (bigramas) frente a otros esquemas de agrupación de términos ofrecieron los mejores resultados comparando los resultados con búsquedas sintácticas

Characterization of Fake News Based on Subjectivity Lexicons

While many works investigate spread patterns of fake news in social networks, we focus on the textual content. Instead of relying on syntactic representations of documents (aka Bag of Words) as many works do, we seek more robust representations that may better differentiate fake from legitimate news. We propose to consider the subjectivity of news under the assumption that the subjectivity levels of legitimate and fake news are significantly different. For computing the subjectivity level of news, we rely on a set subjectivity lexicons for both Brazilian Portuguese and English languages. We then build subjectivity feature vectors for each news article by calculating the Word Mover's Distance (WMD) between the news and these lexicons considering the embedding the news words lie in, in order to analyze and classify the documents. The results demonstrate that our method is robust, especially in scenarios where training and test domains are different.

Multi-objective multi-view based search result clustering using differential evolution framework

Search Results Clustering (SRC) is a well-known problem in the field of information retrieval and refers to the clustering of web-snippets for a given query based on some similarity/dissimilarity measure. In this current study, we have posed Search Results Clustering problem as a multi-view clustering problem and solved it from an optimization point of view. Various views based on syntactic and semantic similarity measures were considered while performing the clustering. In contrast to existing algorithms, three new views based on word mover distance, textual-entailment, and universal sentence encoder, measuring semantics while performing clustering, are incorporated in our framework. Different quality measures computed on clusters generated by different views are optimized simultaneously using multi-objective binary differential evolution (MBDE) framework. MBDE comprises a set of solutions and each solution is composed of two parts corresponding to different views. An agreement index checking the accordance between partitionings of different views is also optimized to obtain a consensus partitioning. The proposed approach is automatic in nature as it is capable of detecting the number of clusters for any query in an automatic way. Experiments are performed on three benchmark multi-view datasets corresponding to web search results and evaluated using well-known F-measure metric. Results obtained illustrate that our approach outperforms state-of-the-art techniques.

Employing word mover's distance for cross‐lingual plagiarized text detection

AbstractWhile the blossom of globalization has achieved many successes, cross‐lingual plagiarism, however, has become popular. Detecting plagiarism across languages requires the capability of comparing semantical similarities between texts of different languages. Previous works rely on massive bilingual resources such as comparable corpus, parallel corpus, and even commercial machine translation as references. However, towards domain‐specific applications, collecting such resources is labor‐intensive and impractical. Also, the absence of interpretability of existing methods lead to the difficulty of investigating retrieval results by humans. Hence, it is imperative to have a resource‐light and interpretable method for cross‐lingual plagiarism detection. In this study, we propose a new detection method, called CL‐WMD, which is built upon word embedding techniques. CL‐WMD requires only a small set of translation pairs to constitute a bilingual reference and calculates semantical distances between texts by word mover's distance, which can provide explicable word alignment information between two compared text spans. Our experiments are conducted under a bilingual scientific publication corpus composed of two typologically diverse languages: English and Chinese. The results demonstrate that CL‐WMD has higher accuracy than most existing methods and achieves better or comparable performance when compared to the translation‐based method in paragraph‐level and sentence‐level plagiarism detection tasks.

Fusion of self-organizing map and granular self-organizing map for microblog summarization

In this paper, we have proposed a fusion of two architectures, self-organizing map and granular self-organizing map (SOM + GSOM), for solving the microblog summarization task where a set of relevant tweets are extracted from the available set of tweets. SOM is used to reduce the available set of tweets to a smaller subset, and GSOM is used for extracting relevant tweets. The fusion of SOM + SOM is also accomplished to illustrate the effectiveness of GSOM over SOM in the second architecture. Moreover, only SOM version is also utilized to illustrate the potentiality of fusion in our proposed approaches. As similarity/dissimilarity measures play major role in any summarization system; therefore, to measure the same between tweets, various measures like word mover distance, cosine distance and Euclidean distance are also explored. The results obtained are evaluated on four datasets related to disaster events using ROUGE measures. Experimental results demonstrate that our best-proposed approach (SOM + GSOM) has obtained $$17\%$$ and $$5.9\%$$ improvements in terms of ROUGE-2 and ROUGE-L scores, respectively, over the existing techniques. The results are also validated using statistical significance t-test.

Identifying click baits using various machine learning and deep learning techniques

In today’s world, most readers prefer to read news online as they get instant access to what is happening right now. Furthermore, personalized recommendations help in keeping users engaged. Along with these virtues, online news has some vices as well. One such vice is the presence of alluring social media posts (tweets) relating to news articles whose sole purpose is to draw the attention of the users rather than directing them to read the actual content. Such posts are referred to as click baits. The objective of this paper is to develop a system which is capable of predicting how likely are the social media posts (tweets) relating to new articles tend to be click baits. GloVe embeddings [Pennington et al. in: Empirical methods in natural language processing (EMNLP), pp 1532–1543, 2014] have been used to represent text data numerically. Various novel features (like Word mover’s distances (Kusner et al. in: Proceedings of the 32nd international conference on international conference on machine learning, ICML’15, vol 37, pp 957–966, 2015), subjectivity, polarity of the tweets and so on) have been engineered. Several machine learning-based models like Logistic Regression, Random Forest, XG-Boost and Light GBM have been trained for classification. Moreover, we have also implemented a few deep learning-based models like Deep Neural Networks and Long Short Term Memory for developing this predictive system.

Combining Distributed Word Representation and Document Distance for Short Text Document Clustering

This paper presents a method for clustering short text documents, such as news headlines, social media statuses, or instant messages. Due to the characteristics of these documents, which are usually short and sparse, an appropriate technique is required to discover hidden knowledge. The objective of this paper is to identify the combination of document representation, document distance, and document clustering that yields the best clustering quality. Document representations are expanded by external knowledge sources represented by a Distributed Representation. To cluster documents, a K-means partitioning-based clustering technique is applied, where the similarities of documents are measured by word mover’s distance. To validate the effectiveness of the proposed method, experiments were conducted to compare the clustering quality against several leading methods. The proposed method produced clusters of documents that resulted in higher precision, recall, F1- score, and adjusted Rand index for both real-world and standard data sets. Furthermore, manual inspection of the clustering results was conducted to observe the efficacy of the proposed method. The topics of each document cluster are undoubtedly reflected by members in the cluster.

Fast paraphrase extraction in Ancient Greek literature

Abstract In this paper, A shorter version of the paper appeared in German in the final report of the Digital Plato project which was funded by the Volkswagen Foundation from 2016 to 2019. [35], [28]. we present a method for paraphrase extraction in Ancient Greek that can be applied to huge text corpora in interactive humanities applications. Since lexical databases and POS tagging are either unavailable or do not achieve sufficient accuracy for ancient languages, our approach is based on pure word embeddings and the word mover’s distance (WMD) [20]. We show how to adapt the WMD approach to paraphrase searching such that the expensive WMD computation has to be computed for a small fraction of the text segments contained in the corpus, only. Formally, the time complexity will be reduced from O ( N · K 3 · log K ) \mathcal{O}(N\cdot {K^{3}}\cdot \log K) to O ( N + K 3 · log K ) \mathcal{O}(N+{K^{3}}\cdot \log K) , compared to the brute-force approach which computes the WMD between each text segment of the corpus and the search query. N is the length of the corpus and K the size of its vocabulary. The method, which searches not only for paraphrases of the same length as the search query but also for paraphrases of varying lengths, was evaluated on the Thesaurus Linguae Graecae® (TLG®) [25]. The TLG consists of about 75 · 10 6 75\cdot {10^{6}} Greek words. We searched the whole TLG for paraphrases for given passages of Plato. The experimental results show that our method and the brute-force approach, with only very few exceptions, propose the same text passages in the TLG as possible paraphrases. The computation times of our method are in a range that allows its application in interactive systems and let the humanities scholars work productively and smoothly.