Word Semantic Similarity Research Articles

Cross-lingual embeddings with auxiliary topic models

Projection-based methods for generating high-quality Cross-Lingual Embeddings (CLEs) have shown state-of-the-art performance in many multilingual applications. Supervised methods that rely on character-level information or unsupervised methods that need only monolingual information are both popular and have their pros and cons. However, there are still problems in terms of the quality of monolingual word embedding spaces and the generation of the seed dictionaries. In this work, we aim to generate effective CLEs with auxiliary Topic Models. We utilize both monolingual and bilingual topic models in the procedure of generating monolingual embedding spaces and seed dictionaries for projection. We present a comprehensive evaluation of our proposed model through the means of bilingual lexicon extraction, cross-lingual semantic word similarity and cross-lingual document classification tasks. We show that our proposed model outperforms existing supervised and unsupervised CLE models built on basic monolingual embedding spaces and seed dictionaries. It also exceeds CLE models generated from representative monolingual topical word embeddings.

Research of Toponyms of the Irkutsk Region Using the Method of Artificial Intelligence

Essential issues of toponymy presuppose studying separate words to reconstruct the denotative meaning of geographical names that were lost in the modern language and to find out how the peculiarities of the local topography, the inhabitants’ activities, etc. are reflected in them. It is possible to solve this kind of problems using intellectual methods of data analysis on the basis of information technologies. However, in scientific literature on toponymy, such methods are practically ignored. The article is devoted to the study of the origin and semantic meanings of geographical names based on finding semantic associates and calculating the semantic similarity of words using the embedding model. According to the proposed method, the origin of some toponyms of the Irkutsk region was determined, their semantic relations were revealed. The dichotomy method was used for toponyms that have two roots in their structure. This made it possible to improve the operation of the model by clarifying the morphemic composition of the original word. The method of word transformation was used to determine the etymology of the toponym «Moscow». We have received new versions of the origin of the toponym. It is shown that the application of the methods based on distributive semantics and vector representation of words, obtained on the basis of large arrays of text data, significantly expands the possibilities of research in the field of determining the origin of toponyms and clarifying their meaning.

Knowledge-based sentence semantic similarity: algebraical properties

Determining the extent to which two text snippets are semantically equivalent is a well-researched topic in the areas of natural language processing, information retrieval and text summarization. The sentence-to-sentence similarity scoring is extensively used in both generic and query-based summarization of documents as a significance or a similarity indicator. Nevertheless, most of these applications utilize the concept of semantic similarity measure only as a tool, without paying importance to the inherent properties of such tools that ultimately restrict the scope and technical soundness of the underlined applications. This paper aims to contribute to fill in this gap. It investigates three popular WordNet hierarchical semantic similarity measures, namely path-length, Wu and Palmer and Leacock and Chodorow, from both algebraical and intuitive properties, highlighting their inherent limitations and theoretical constraints. We have especially examined properties related to range and scope of the semantic similarity score, incremental monotonicity evolution, monotonicity with respect to hyponymy/hypernymy relationship as well as a set of interactive properties. Extension from word semantic similarity to sentence similarity has also been investigated using a pairwise canonical extension. Properties of the underlined sentence-to-sentence similarity are examined and scrutinized. Next, to overcome inherent limitations of WordNet semantic similarity in terms of accounting for various Part-of-Speech word categories, a WordNet “All word-To-Noun conversion” that makes use of Categorial Variation Database (CatVar) is put forward and evaluated using a publicly available dataset with a comparison with some state-of-the-art methods. The finding demonstrates the feasibility of the proposal and opens up new opportunities in information retrieval and natural language processing tasks.

An Artificial Intelligence Approach for Word Semantic Similarity Measure of Hindi Language

AI combined with NLP techniques has promoted the use of Virtual Assistants and have made people rely on them for many diverse uses. Conversational Agents are the most promising technique that assists computer users through their operation. An important challenge in developing Conversational Agents globally is transferring the groundbreaking expertise obtained in English to other languages. AI is making it possible to transfer this learning. There is a dire need to develop systems that understand secular languages. One such difficult language is Hindi, which is the fourth most spoken language in the world. Semantic similarity is an important part of Natural Language Processing, which involves applications such as ontology learning and information extraction, for developing conversational agents. Most of the research is concentrated on English and other European languages. This paper presents a Corpus-based word semantic similarity measure for Hindi. An experiment involving the translation of the English benchmark dataset to Hindi is performed, investigating the incorporation of the corpus, with human and machine similarity ratings. A significant correlation to the human intuition and the algorithm ratings has been calculated for analyzing the accuracy of the proposed similarity measures. The method can be adapted in various applications of word semantic similarity or module for any other language.

Building Arabic Paraphrasing Benchmark based on Transformation Rules

Measuring semantic similarity between short texts is an important task in many applications of natural language processing, such as paraphrasing identification. This process requires a benchmark of sentence pairs that are labeled by Arab linguists and considered a standard that can be used by researchers when evaluating their results. This research describes an Arabic paraphrasing benchmark to be a good standard for evaluation algorithms that are developed to measure semantic similarity for Arabic sentences to detect paraphrasing in the same language. The transformed sentences are in accordance with a set of rules for Arabic paraphrasing. These sentences are constructed from the words in the Arabic word semantic similarity dataset and from different Arabic books, educational texts, and lexicons. The proposed benchmark consists of 1,010 sentence pairs wherein each pair is tagged with scores determining semantic similarity and paraphrasing. The quality of the data is assessed using statistical analysis for the distribution of the sentences over the Arabic transformation rules and exploration through hierarchical clustering (HCL). Our exploration using HCL shows that the sentences in the proposed benchmark are grouped into 27 clusters representing different subjects. The inter-annotator agreement measures show a moderate agreement for the annotations of the graduate students and a poor reliability for the annotations of the undergraduate students.

Semantic Similarity of Inverse Morpheme Words Based on Word Embedding

Inverse morpheme words are compound words that have the same morphemes but are arranged in the opposite order. The majority of related works on the subject have focused on a narrow investigation of dictionary definitions, with few studies based on large-scale corpora. We used the People’s Daily corpus (1946–2017) to add and delete words from a base list and obtained a word list of 668 pairs of inverse morpheme words. Furthermore, the cosine similarity is computed by using word embedding based on the distributed representation, and the Pearson correlation coefficient between it and the manually annotated value is 0.907, indicating that this method can measure the semantic similarity of inverse morpheme words very close to human judgment. We also discovered that 76% of inverse morpheme words have a cosine similarity of 0.4 or higher and that word formation, part-of-speech, and frequency all have an impact on semantic similarity.

MultiGBS: A multi-layer graph approach to biomedical summarization

Automatic text summarization methods generate a shorter version of the input text to assist the reader in gaining a quick yet informative gist. Existing text summarization methods generally focus on a single aspect of text when selecting sentences, causing the potential loss of essential information. In this study, we propose a domain-specific method that models a document as a multi-layer graph to enable multiple features of the text to be processed at the same time. The features we used in this paper are word similarity, semantic similarity, and co-reference similarity, which are modelled as three different layers. The unsupervised method selects sentences from the multi-layer graph based on the MultiRank algorithm and the number of concepts. The proposed MultiGBS algorithm employs UMLS and extracts the concepts and relationships using different tools such as SemRep, MetaMap, and OGER. Extensive evaluation by ROUGE and BERTScore shows increased F-measure values.

AWN-similarity: Towards developing free open-source frameworks for measuring Arabic semantic similarity under Windows / Linux operating systems

Arabic is a highly systematic language where its words exhibit elegant and rigorous logic. The field of Arabic word semantic similarity becomes more challenging due to its higher complexity and subtlety. This research is concerned with investigating the development of free open-source frameworks containing packages to calculate the semantic similarity between two Arabic words or concepts. These packages are known as AWN-ConceptSimilarity and AWN-WordSimilarity. The developed packages implement seven semantic similarity algorithms. One of these algorithms was proposed for Arabic and the rest were proposed for English where successfully adapted to Arabic using an Arabic lexical database, Arabic wordnet. The functionality of the developed packages is validated using two-word similarity benchmarks datasets previously produced for Arabic. The results of the validation process indicate that the developed frameworks represent an important contribution to the Arabic semantic similarity field. Moreover, the developed packages are reliable to use and embed them with Arabic researchers' projects for improving or comparing their methodologies.

AWN-similarity: Towards developing free open-source frameworks for measuring Arabic semantic similarity under Windows / Linux operating systems

Arabic is a highly systematic language where its words exhibit elegant and rigorous logic. The field of Arabic word semantic similarity becomes more challenging due to its higher complexity and subtlety. This research is concerned with investigating the development of free open-source frameworks containing packages to calculate the semantic similarity between two Arabic words or concepts. These packages are known as AWN-ConceptSimilarity and AWN-WordSimilarity. The developed packages implement seven semantic similarity algorithms. One of these algorithms was proposed for Arabic and the rest were proposed for English where successfully adapted to Arabic using an Arabic lexical database, Arabic wordnet. The functionality of the developed packages is validated using two-word similarity benchmarks datasets previously produced for Arabic. The results of the validation process indicate that the developed frameworks represent an important contribution to the Arabic semantic similarity field. Moreover, the developed packages are reliable to use and embed them with Arabic researchers' projects for improving or comparing their methodologies.

Modeling Literary Criticism: How to Do It and How to Teach It to Humans and Machines

Modeling Literary Criticism: How to Do It and How to Teach It to Humans and Machines

Applying Clustering and Topic Modeling to Automatic Analysis of Citizens’ Comments in EGovernment

The paper proposes an approach to analyze citizens' comments in e-government using topic modeling and clustering algorithms. The main purpose of the proposed approach is to determine what topics are the citizens' commentaries about written in the e-government environment and to improve the quality of e-services. One of the methods used to determine this is topic modeling methods. In the proposed approach, first citizens' comments are clustered and then the topics are extracted from each cluster. Thus, we can determine which topics are discussed by citizens. However, in the usage of clustering and topic modeling methods appear some problems. These problems include the size of the vectors and the collection of semantically related of documents in different clusters. Considering this, the semantic similarity of words is used in the approach to reduce measure. Therefore, we only save one of the words that are semantically similar to each other and throw the others away. So, the size of the vector is reduced. Then the documents are clustered and topics are extracted from each cluster. The proposed method can significantly reduce the size of a large set of documents, save time spent on the analysis of this data, and improve the quality of clustering and LDA algorithm.

Duration of verbal fluency inter‐word intervals and risk of cognitive impairment

AbstractBackgroundVerbal fluency tasks have widely recognized value for dementia diagnosis. Some studies suggest that detailed analysis of verbal fluency word lists may aid in prediction of future cognitive decline or dementia.MethodWe identified 701 stroke‐free participants in the Reasons for Geographic and Racial Differences in Stroke (REGARDS) study, all of whom performed in the normal range on a cognitive screening test at baseline, but who suffered from acute or progressive incident cognitive impairment (ICI) during subsequent years. These cases were matched with 701 controls who did not suffer subsequent cognitive decline. Baseline letter F and animal fluency recordings were automatically transcribed using Amazon Web Services’ Transcribe , then manually corrected using a Matlab‐based graphical transcription tool for precise capture of utterance latencies and durations. We performed two statistical analyses on the durations of inter‐word intervals (IWI). The first employed mixed‐effects linear models with IWI as the dependent variable and several lexical (e.g., frequency, edit distance from previous word, etc.) and demographic covariates. The second employed Cox proportional hazards (CPH) models, with time to ICI as the dependent variable and summary measures of IWI (sum of z‐scores) as key predictors of interest.ResultMixed‐effects models show that semantic similarity of consecutive words and overall within‐task latency of words are the strongest predictors of IWI, but that future progressive cognitive decline is associated with longer IWI on the animal fluency task. CPH models show significant independent contributions of animal fluency raw score and IWI summary score. Slowing of word retrieval reflected by the latter score was associated with greater risk for cognitive decline.ConclusionDetailed analysis of IWI measured during fluency tasks may enhance the prognostic value of verbal fluency tests. The effect appears to be strongest with animal fluency predicting progressive decline.

Improving biterm topic model with word embeddings

As one of the fundamental information extraction methods, topic model has been widely used in text clustering, information recommendation and other text analysis tasks. Conventional topic models mainly utilize word co-occurrence information in texts for topic inference. However, it is usually hard to extract a group of words that are semantically coherent and have competent representation ability when the models applied into short texts. It is because the feature space of the short texts is too sparse to provide enough co-occurrence information for topic inference. The continuous development of word embeddings brings new representation of words and more effective measurement of word semantic similarity from concept perspective. In this study, we first mine word co-occurrence patterns (i.e., biterms) from short text corpus and then calculate biterm frequency and semantic similarity between its two words. The result shows that a biterm with higher frequency or semantic similarity usually has more similar words in the corpus. Based on the result, we develop a novel probabilistic topic model, named Noise Biterm Topic Model with Word Embeddings (NBTMWE). NBTMWE extends the Biterm Topic Model (BTM) by introducing a noise topic with prior knowledge of frequency and semantic similarity of biterm. NBTMWE shows the following advantages compared with BTM: (1) It can distinguish meaningful latent topics from a noise topic which consists of some common-used words that appear in many texts of the dataset; (2) It can promote a biterm’s semantically related words to the same topic during the sampling process via generalized $P\acute {o}lya$ Urn (GPU) model. Using auxiliary word embeddings trained from a large scale of corpus, we report the results testing on two short text datasets (i.e., Sina Weibo and Web Snippets). Quantitatively, NBTMWE outperforms the state-of-the-art models in terms of coherence, topic word similarity and classification accuracy. Qualitatively, each of the topics generated by NBTMWE contains more semantically similar words and shows superior intelligibility.

Understanding the spatial dimension of natural language by measuring the spatial semantic similarity of words through a scalable geospatial context window

Measuring the semantic similarity between words is important for natural language processing tasks. The traditional models of semantic similarity perform well in most cases, but when dealing with words that involve geographical context, spatial semantics of implied spatial information are rarely preserved. Geographic information retrieval (GIR) methods have focused on this issue; however, they sometimes fail to solve the problem because the spatial and textual similarities of words are considered and calculated separately. In this paper, from the perspective of spatial context, we consider the two parts as a whole—spatial context semantics, and we propose a method that measures spatial semantic similarity using a sliding geospatial context window for geo-tagged words. The proposed method was first validated with a set of simulated data and then applied to a real-world dataset from Flickr. As a result, a spatial semantic similarity model at different scales is presented. We believe this model is a necessary supplement for traditional textual-language semantic analyses of words obtained by word-embedding technologies. This study has the potential to improve the quality of recommendation systems by considering relevant spatial context semantics, and benefits linguistic semantic research by emphasising the spatial cognition among words.

Representation of associative and affective semantic similarity of abstract words in the lateral temporal perisylvian language regions

The examination of semantic cognition has traditionally identified word concreteness as well as valence as two of the principal dimensions in the representation of conceptual knowledge. More recently, corpus-based vector space models as well as graph-theoretical analysis of large-scale task-related behavioural responses have revolutionized our insight into how the meaning of words is structured. In this fMRI study, we apply representational similarity analysis to investigate the conceptual representation of abstract words. Brain activity patterns were related to a cued-association based graph as well as to a vector-based co-occurrence model of word meaning. Twenty-six subjects (19 females and 7 males) performed an overt repetition task during fMRI. First, we performed a searchlight classification procedure to identify regions where activity is discriminable between abstract and concrete words. These regions were left inferior frontal gyrus, the upper and lower bank of the superior temporal sulcus bilaterally, posterior middle temporal gyrus and left fusiform gyrus. Representational Similarity Analysis demonstrated that for abstract words, the similarity of activity patterns in the cortex surrounding the superior temporal sulcus bilaterally and in the left anterior superior temporal gyrus reflects the similarity in word meaning. These effects were strongest for semantic similarity derived from the cued association-based graph and for affective similarity derived from either of the two models. The latter effect was mainly driven by positive valence words. This research highlights the close neurobiological link between the information structure of abstract and affective word content and the similarity in activity pattern in the lateral and anterior temporal language system.

A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages

Measuring the semantic similarity and relatedness of words is important for many natural language processing tasks. Although distributional semantic models designed for this task have many different parameters, such as vector similarity measures, weighting schemes and dimensionality reduction techniques, there is no truly comprehensive study simultaneously evaluating these parameters while also analysing the differences in the findings for multiple languages. We would like to address this gap with our systematic study by searching for the best configuration in the creation and comparison of feature vectors in distributional semantic models for English, Spanish and Hungarian separately, and then comparing our findings across these languages. During our extensive analysis we test a large number of possible settings for all parameters, with more than a thousand novel variants in case of some of them. As a result of this we were able to find such configurations that significantly outperform conventional configurations and achieve state-of-the-art results.

A Natural Language Process-Based Framework for Automatic Association Word Extraction

Word association, revealing mental representations and connections of human, has been widely studied in psychology. However, the scale of available associative cue-response words is severely restricted due to the traditional manually collecting methodology. Meanwhile, with the tremendous success in Natural Language Process (NLP) tasks, an extremely large amount of plain texts can be easily acquired. This suggests an insight about the potential to find association words automatically from the text corpus instead of manually collection. As an original attempt, this paper takes a small step toward proposing a deep learning based framework for automatic association word extraction. The framework mainly consists of two stages of association word detection and machine association network construction. In particular, attention mechanism based Reading Comprehension (RC) algorithm is explored to find valuable association words automatically. To validate the value of the extracted association words, the correlation coefficient between semantic similarities of machine and human association words is introduced as an effective measurement for evaluating association consistence. The experiments are conducted on two text datasets from which together about 20k association words, more than the existing largest human association word dataset, are finally derived. The experiment further verifies that the machine association words are generally consistent with human association words with respect to semantic similarity, which highlights the promising utilization of the machine association words in the future researches of both psychology and NLP.

METHOD FOR AUTOMATIC IDENTIFICATION OF SEMANTICALLY SIMILAR FRAGMENTS OF NEWS TEXTS

Depending on the ambiguity and synonymy at all language levels, the identification of semantic similarity of linguistic units is the challenging task of semantic analysis of text information contained in news reports. The extraction of semantically similar fragments of texts or paraphrases is an up-to-date problem in fields of science such as semantic information retrieval, information extraction, machine translation, detection of copyright infringements, etc. and is widely used in rewriting. The article analyzes the main problems of rewriting, in particular, the paraphrasing of syntactic text units keeping the sense load. The modern methods for identification of semantic similarity of words, their advantages and disadvantages are considered. Based on the use of WordNet and the developed syntactic rules that store information about the grammatical characteristics of words, a method for automatic identification of synonymous fragments of news texts is proposed. The advantage of this method is that both the grammatical structure of the language and the meaning of words (using WordNet) are analyzed. The experimental corpus is represented by news texts from Reuters news agency, BBC World News and CNN services. The proposed method for identifying semantically similar text fragments allows defining the common information space of current news and can be used to effectively identify related texts in information retrieval, expert, analytical information and rewriting systems. The automatic identification of semantic similarity could be implemented in automated construction of ontologies, in expansion of existing and creation of new thesauri.

СИСТЕМАТИЗАЦИЯ СМЫСЛОВЫХ ОТНОШЕНИЙ ПРОИЗВОДЯЩИХ ПОЛИСЕМАНТОВ ПРИ ОТРАЖЕННЫХ СИНОНИМАХ

The article is devoted to identifying and systematizing semantic relations between productive words with reflected synonyms appearing as a result of figurative word-formative motivation. The originality of the study is conditioned by the fact that the semantic similarity of productive words is explicated at the level of primary lexico-semantic variants. The findings indicate that reflected synonymy and figurative word-formative motivation are determined by semantic interrelation of productive words, which is implemented through logical or associative relations.

Analysing political events on Twitter

Recently, political events, such as elections, have raised a lot of discussions on social media networks, in particular, Twitter. This brings new opportunities for social scientists to address social science tasks, such as understanding what communities said or identifying whether a community has an influence on another. However, identifying these communities and extracting what they said from social media data are challenging and non-trivial tasks. We aim to make progress towards understanding 'who' (i.e. communities) said 'what' (i.e. discussed topics) and 'when' (i.e. time) during political events on Twitter. While identifying the 'who' can benefit from Twitter user community classification approaches, 'what' they said and 'when' can be effectively addressed on Twitter by extracting their discussed topics using topic modelling approaches that also account for the importance of time on Twitter. To evaluate the quality of these topics, it is necessary to investigate how coherent these topics are to humans. Accordingly, we propose a series of approaches in this thesis. First, we investigate how to effectively evaluate the coherence of the topics generated using a topic modelling approach. The topic coherence metric evaluates the topical coherence by examining the semantic similarity among words in a topic. We argue that the semantic similarity of words in tweets can be effectively captured by using word embeddings trained using a Twitter background dataset. Through a user study, we demonstrate that our proposed word embedding-based topic coherence metric can assess the coherence of topics like humans [1, 2]. In addition, inspired by the precision at k metric, we propose to evaluate the coherence of a topic model (containing many topics) by averaging the top-ranked topics within the topic model [3]. Our proposed metrics can not only evaluate the coherence of topics and topic models, but also can help users to choose the most coherent topics. Second, we aim to extract topics with a high coherence from Twitter data. Such topics can be easily interpreted by humans and they can assist to examine 'what' has been discussed and 'when'. Indeed, we argue that topics can be discussed in different time periods (see [4]) and therefore can be effectively identified and distinguished by considering their time periods. Hence, we propose an effective time-sensitive topic modelling approach by integrating the time dimension of tweets (i.e. 'when') [5]. We show that the time dimension helps to generate topics with a high coherence. Hence, we argue that 'what' has been discussed and 'when' can be effectively addressed by our proposed time-sensitive topic modelling approach. Next, to identify 'who' participated in the topic discussions, we propose approaches to identify the community affiliations of Twitter users, including automatic ground-truth generation approaches and a user community classification approach. We show that the mentioned hashtags and entities in the users' tweets can indicate which community a Twitter user belongs to. Hence, we argue that they can be used to generate the ground-truth data for classifying users into communities. On the other hand, we argue that different communities favour different topic discussions and their community affiliations can be identified by leveraging the discussed topics. Accordingly, we propose a Topic-Based Naive Bayes (TBNB) classification approach to classify Twitter users based on their words and discussed topics [6]. We demonstrate that our TBNB classifier together with the ground-truth generation approaches can effectively identify the community affiliations of Twitter users. Finally, to show the generalisation of our approaches, we apply our approaches to analyse 3.6 million tweets related to US Election 2016 on Twitter [7]. We show that our TBNB approach can effectively identify the 'who', i.e. classify Twitter users into communities. To investigate 'what' these communities have discussed, we apply our time-sensitive topic modelling approach to extract coherent topics. We finally analyse the community-related topics evaluated and selected using our proposed topic coherence metrics. Overall, we contribute to provide effective approaches to assist social scientists towards analysing political events on Twitter. These approaches include topic coherence metrics, a time-sensitive topic modelling approach and approaches for classifying the community affiliations of Twitter users. Together they make progress to study and understand the connections and dynamics among communities on Twitter. Supervisors : Iadh Ounis, Craig Macdonald, Philip Habel The thesis is available at http://theses.gla.ac.uk/41135/