Vector Space Model Research Articles

Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms.

Most of the modern natural language processing (NLP) techniques are based on the vector space models of language, in which each word is represented by a vector in a high dimensional space. One of the earliest successes was demonstrated by the four-term analogical reasoning task: what is to C as B is to A? The trained word vectors form "parallelograms" representing the quadruple of words in analogy. This discovery in NLP offers us insight into our understanding of human semantic representation of words via analogical reasoning. Despite successful applications of the large-scale language models, it has not been fully understood why such parallelograms emerge by learning through natural language data. As the vector space model is not optimized to form parallelograms, the key structure related to geometric shapes of word vectors is expected to be in the data, rather than the models. In the present article, we test our hypothesis that such parallelogram arrangement of word vectors readily exists in the co-occurrence statistics of language. Our approach focuses more on the data itself, and it is different from the existing theoretical approach trying to find the mechanism of parallelogram formation in the algorithms and/or vector arithmetic operations on word vectors. First, our analysis suggested that analogical reasoning is possible by decomposition of the bigram co-occurrence matrix. Second, we demonstrated the formation of a parallelepiped, a more structured geometric object than a parallelogram, by creating a small artificial corpus and its word vectors. With these results, we propose a refined form of distributional hypothesis pointing out an isomorphism between a sort of symmetry or exchangeability and word co-occurrence statistics.

Correction to: Measuring the semantic headedness of English blends with token-based semantic vector space modeling: a corpus-based study

Correction to: Measuring the semantic headedness of English blends with token-based semantic vector space modeling: a corpus-based study

Measuring the semantic headedness of English blends with token-based semantic vector space modeling: a corpus-based study

Abstract This article analyzes the semantic headedness of English blends with distributional semantics methods. The semantic head of a blend is the source word that transfers its semantic information to the blend as a whole. For example, a sitcom is a kind of comedy. But is FedEx a kind of express, and is wi-fi a kind of fidelity? We use corpus data and token-based semantic vector space modeling in order to address these questions. Specifically, we investigate whether Plag’s ternary division of endocentric, exocentric, and coordinative compounds based on semantic headedness can also be applied to English blends, and whether the general tendency of semantic right-headedness can be observed for all three subtypes. We analyze a dataset of fifty-five blends and their respective source words, using data from the Corpus of Contemporary American English and the English Web Corpus 2021. We measure the degree of semantic similarity between each blend and its two source words. The results show that for most endocentric blends, the hypothesis of semantic right-headedness holds true. At the same time, exocentric blends and coordinative blends are shown to behave differently. We conclude that Plag’s classification offers a useful point of departure for the semantic analysis of blends and that distributional semantics methods can provide new insights into their semantic behavior.

Subword Representations Successfully Decode Brain Responses to Morphologically Complex Written Words

Abstract This study extends the idea of decoding word-evoked brain activations using a corpus-semantic vector space to multimorphemic words in the agglutinative Finnish language. The corpus-semantic models are trained on word segments, and decoding is carried out with word vectors that are composed of these segments. We tested several alternative vector-space models using different segmentations: no segmentation (whole word), linguistic morphemes, statistical morphemes, random segmentation, and character-level 1-, 2- and 3-grams, and paired them with recorded MEG responses to multimorphemic words in a visual word recognition task. For all variants, the decoding accuracy exceeded the standard word-label permutation-based significance thresholds at 350–500 ms after stimulus onset. However, the critical segment-label permutation test revealed that only those segmentations that were morphologically aware reached significance in the brain decoding task. The results suggest that both whole-word forms and morphemes are represented in the brain and show that neural decoding using corpus-semantic word representations derived from compositional subword segments is applicable also for multimorphemic word forms. This is especially relevant for languages with complex morphology, because a large proportion of word forms are rare and it can be difficult to find statistically reliable surface representations for them in any large corpus.

Content Classification of the Official Website of the Ministry of Foreign Affairs of the Republic of Indonesia (MoFA RI) using Vector Space Model (VSM)

The official website of the Ministry of Foreign Affairs of the Republic of Indonesia (MoFA RI) is an important platform for disseminating information to a diverse audience. Efficiently categorizing the vast amount of content available on the website is essential for enhancing user experience and optimizing information retrieval. These categories will also become an identifier and topic classification based on the content inside the article. This study presents a systematic approach to content classification of the Official Website of the Ministry of Foreign Affairs of the Republic of Indonesia (MoFA RI) using the Vector Space Model (VSM). The methodology involves preprocessing the text data, constructing a term-document matrix, and implementing cosine similarity to measure the relevance of documents to predefined categories. The study demonstrates the effectiveness of VSM in accurately classifying content, thus facilitating streamlined access to information for users navigating the website. Furthermore, the findings offer insights into enhancing the organization and accessibility of governmental online platforms, contributing to improved user experience and information dissemination.

Prototipe Sistem Rekomendasi Film Indonesia Menggunakan Pendekatan Content Based Filtering dan Metode Vector Space Model

Film represents a combination of narrative and cinematographic aspects in an audio-visual form. Films also provide an engaging visual, auditory, and emotional experience. The film industry in Indonesia has shown significant growth. In 2015, cinema audiences numbered 16.2 million, increasing to 51.2 million in 2019. The COVID-19 pandemic caused a drastic decline in the number of viewers, with only 4.5 million in 2021. However, the industry quickly recovered, reaching 54.07 million viewers in 2022 and 55 million in 2023, and is projected to reach 60 million in 2024. With the increasing availability of films and streaming platforms, finding movies that match user preferences often becomes a challenge. This study proposes a film recommendation system using the Content-Based Filtering method and the Vector Space Model to help users more easily find movies that match their preferences. Content-Based Filtering recommends films based on content similarity, while the Vector Space Model measures content similarity using metrics such as cosine similarity or euclidean distance. Based on experiments using 20 sample films, the results show that the designed system can provide accurate and precise movie recommendations based on keywords provided by users. The film "Pengabdi Setan 2: Communion" has the highest similarity score of 0.3018, followed by "Nana," "Menjelang Magrib," "Jailangkung: Sandekala," and "Qorin," with similarity scores of 0.0865, 0.0138, 0.0136, and 0.0125, respectively

Exploring The Impact of Stemming on Text Topic-Based Classification Accuracy

Text classification attempts to assign written texts to specific group types that share the same linguistic features. One class of features that have been widely employed for a wide range of classification tasks is lexical features. This study explores the impact of stemming on text classification using lexical features. To explore, this study is based on a corpus of thirty texts written by six authors with topics that focus on politics, history, science, prose, sport, and food. These texts are stemmed using a light stemming algorithm. In order to classify these texts according to the topic by means of lexical features, linear hierarchical clustering and non-linear clustering (SOM) is carried out on the stemmed and unstemmed texts. Although both clustering methods are able to classify texts by topic with two models produce accurate and stable results, the results suggest that the impact of a light stemming on the accuracy of text classification by topic is ineffectual. The accuracy is neither increased nor decreased on the stemmed texts, whereby the stemming algorithm helped reducing the dimensionality of feature vector space model.

Various Approaches for Software vulnerability Assessment and Risk Identification using soft computing Techniques

The detection of software flaws is a viable technique for enhancing the quality of technology and testing management by providing rapid detection of deficiency simulation models until the actual testing phase starts. These prediction outcomes help designers of technology effectively devote their available resources to components that are more vulnerable to deficiencies. In this research investigation, a software bug prediction model is proposed using Deep Learning (DL) approach. The Recurrent Neural Network (RNN) is used for classification of source code including numerous soft computing techniques. Numerous preprocessing and data filtration techniques have been carried out for data balancing and normalization. The Term Frequency and Inverse Document Frequency (TF-IDF) and relation features extraction techniques is used to generate Vector Space Model (VSM). The classification has been done using RNN on both training and validation dataset. The evaluation of performance of proposed work is performed using various real-time and synthetic accessible databases. It is observed from the experimental results that the proposed framework performs better when different evaluated with different datasets

(Dis)similarities between non-symbolic and symbolic number representations: Insights from vector space models

Empirical evidence in support of a shared system for non-symbolic and symbolic number processing has been inconclusive. The current study aims to address this question in a novel way, specifically by testing whether the efficient coding principle based on co-occurrence of number symbols in natural language holds for both non-symbolic and symbolic number processing. The efficient coding principle postulates that perception is optimized when stimuli frequently co-occur in a natural environment. We hypothesized that both numerical ratios and co-occurrence frequencies of symbolic numbers would significantly influence participants' performance on a non-symbolic and symbolic number comparison task. To test this hypothesis, we employed latent semantic analysis on a TASA corpus to quantify number co-occurrence in natural language and calculate language similarity estimates. We engaged 73 native English speakers (mean age = 19.36, standard deviation = 1.83) with normal or corrected vision and no learning disorders in a number comparison task involving non-symbolic (dot arrays) and symbolic stimuli (Arabic numerals and English number words). Results showed that numerical ratios significantly predicted participants' performances across all number formats (ps < 0.001). Language similarity estimates derived from everyday language also predicted performance on the non-symbolic task and the symbolic task involving number words (ps < 0.007). Our results highlight the complex nature of numerical processing, pointing to the co-occurrence of number symbols in natural language as an auxiliary factor in understanding the shared characteristics between non-symbolic and symbolic number representations. Given that our study focused on a limited number range (5 to 16) and a specific task type, future studies should explore a wider range of tasks and numbers to further test the role of the efficient coding principle in number processing.

A modified Vector Space Model for semantic information retrieval

In this research, we present a modified Vector Space Model which focuses on the semantic relevance of words for retrieving documents. The modified VSM resolves the problem of the classical model performing only lexical matching of query terms to document terms for retrievals. This problem also restricts the classical model from retrieving documents that do not have exact match of query terms even if they are semantically relevant to the query. In the modified model, we introduced a Query Relevance Update technique, which pads the original query set with semantically relevant document terms for optimised semantic retrieval results. The modified model also includes a novel tf−p which replaces the tf−idf technique of the classical VSM, which is used to compute the Term Frequency weights. The replacement of the tf−idf resolves the problem of the classical model penalising terms that occur across documents with the assumption that they are stop words, which in practice, there are usually such words which carry relevant semantic information for documents’ retrieval. We also extended the cosine similarity function with a proportionality weight pqd, which moderates biases for high frequency of terms in longer documents. The pqd ensures that the frequency of query terms including the updated ones are accounted for in proportionality with documents size for the overall ranking of documents. The simulated results reveal that, the modified VSM does achieve semantic retrieval of documents beyond lexical matching of query and document terms.

Medical Record Document Search with TF-IDF and Vector Space Model (VSM)

The growth of medical record documents is increasing over time, and the various types of diseases and therapies needed are increasing. However, this has not been followed by an effective and efficient search process. This study aims to deal with search problems that often take a long time with search results that are not necessarily as expected by building a search model for medical record documents using the vector space model (VSM) and TF-IDF methods. The VSM method allows retrieval of results that are not the same as the search queries entered by the user but are expected to provide still results relevant to the user's desired needs. The model development process was taken based on the data in the FS_ANAMNESA and FS_DIAGNOSA columns, followed by preprocessing, which consists of deleting blank lines, lowercase, removing punctuation marks, HTML tags, stop words, excess spaces between words, and normalizing typo words, then forming a TF-IDF matrix based on the frequency of occurrence of each word feature, and followed by the calculation of the similarity value of the search query compared to medical record documents based on the cosine similarity formula. The retrieval results were all columns of each existing medical record document and were sorted based on 10 rows with the highest similarity value. The model evaluation results were based on 1000 medical record documents and tested with 20 search queries in this study, which gave an average precision value of 0.548 and an average recall value of 0.796.

Automatic Enemy Identification—Are We There Yet?

Enemy items refer to any two items that should not appear on the same test form (Weir, 2019). Accurately identifying enemy pairs is critical for ensuring the quality and fairness of exams, but it can also be challenging and time-consuming given the large number of possible item pairs in the exam item bank. Various enemy identification approaches have been explored to automate or semi-automate this task. In this process, the critical component is the encoding technique. The better the encoding technique captures the meaning of the sentences, the more accurate the similarity index and enemy classification results will be. This study focuses on evaluating the performance of a transformer-based model against the results from a string-based vector-space model (VSM) encoding technique under different research conditions for multiple-choice and multiple-response items used in a foundational IT certification exam. The results suggest that when using sufficient representative training data and conducting fine-tuning, the transformer-based model significantly outperforms the VSM model for enemy identification.

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461, a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461. While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Research and Development of Digital Work Ticket Information Sharing System Based on Mathematical Differential Improved Vector Space Model

In the traditional production business operation and maintenance maintenance involves the work ticket filled by hand, easy to produce errors, and need to review, approval and other operations, low efficiency. The author proposes a digital work ticket sharing system based on mathematical differential improved vector space model, through the integration with PMS2.0 system and security risk management and control platform, achieve mobile office management away from WEB, in terms of functions, it realizes all the functions of the work ticket module in the PMS2.0 system, and integrates various requirements of the on-site paper work ticket, through the application of digital work ticket, real-time data collection and comprehensive sharing of the whole process and business of work ticket are realized, at the same time, according to the requirements of security management and control, data fusion with the security risk management and control platform is realized to improve work efficiency and security sharing.

Implementation of the Vector Space Model Algorithm for Document Searching in Material Test Monitoring Applications (Case Study: BPJN Jayapura Lab)

The material test monitoring application built in 2021 aims to facilitate BPJN Jayapura's work in entering administrative data files to facilitate the material test monitoring process carried out from the process stage to the material test results stage. However, in using it for 2 years, the application has had problems. The application is unable to search for documents stored in the database so the search is carried out one by one. This makes it difficult for users because there are more than 30 documents. Therefore, this research will focus on creating a document search engine (search engine) using the Vector Space Model method. This algorithm will measure the level of similarity of documents to the keywords or queries entered by the user. This research implements the Vector Space Model method to make it easier to search for documents in the application. In carrying out this research, the Software Development Life Cycle methodology will be used as a method for creating the search engine using the Unified Modeling Language version 2.5.1 modeling tool and the PHP programming language. Based on the tests that have been carried out, the Vector Space Model method has been successfully implemented and is able to function as a search engine with a precision level of 0.83 and a recall of 1 for searching 40 documents as samples. Keywords: Document search; Jayapura; vector space model

Quantum projections on conceptual subspaces: A deeper dive into methodological challenges and opportunities.

In alignment with the distributional hypothesis of language, the work "Quantum Projections on Conceptual Subspaces" (Martínez-Mingo A, Jorge-Botana G, Martinez-Huertas JÁ, et al. Quantum projections on conceptual subspaces. Cogn Syst Res 2023; 82: 101154) proposed a methodology for generating conceptual subspaces from textual information based on previous work (Martinez-Mingo A, Jorge-Botana G and Olmos R. Quantum approach for similarity evaluation in LSA vector space models. 2020). These subspaces enable the utilization of the quantum model of similarity put forth by Pothos and Busemeyer (Pothos E, Busemeyer J. A quantum probability explanation for violations of symmetry in similarity judgments. In Proceedings of the annual meeting of the cognitive science society, 2011, Vol. 33, No. 33), allowing for the empirical examination of the violations of assumptions concerning symmetry and triangular inequality (Tversky A. Features of similarity. Psychol Rev 1977; 84: 327-352; Yearsley JM, Barque-Duran A, Scerrati E, et al. The triangle inequality constraint in similarity judgments. Prog Biophys Mol Biol 2017; 130: 26-32), as well as the diagnosticity effect (Tversky A. Features of similarity. Psychol Rev 1977; 84: 327-352; Yearsley JM, Pothos EM, Barque-Duran A, et al. Context effects in similarity judgments. J Exp Psychol Gen 2022; 151: 711-717), within a data-driven environment. These psychological biases, deeply studied by authors such as Tversky and Kahneman, inform us about the limitations of modeling psychological similarity measures using tools from classical geometry. This commentary aims to offer methodological clarifications, discuss theoretical and practical implications, and speculate on future directions in this field of research. Concretely, it aims to propose the use of different contours (conceptual or contextual) to generate the subspaces, which lead to subspaces of terms or contexts. Once these contours are defined, a differentiation is proposed between Aggregated Terms Subspaces (ATSs), Aggregated Contexts Subspaces (ACSs), and Aggregated Features Subspaces (AFSs) depending on whether we define the subspaces by grouping the terms or contexts within the contour, or from the latent dimensions of the semantic space obtained in the contour window. Finally, new data is provided on the violation of the triangular inequality assumption through the application of the quantum similarity model to ATSs.

Uncertainty in Visual Generative AI

Recently, generative artificial intelligence (GAI) has impressed the world with its ability to create text, images, and videos. However, there are still areas in which GAI produces undesirable or unintended results due to being “uncertain”. Before wider use of AI-generated content, it is important to identify concepts where GAI is uncertain to ensure the usage thereof is ethical and to direct efforts for improvement. This study proposes a general pipeline to automatically quantify uncertainty within GAI. To measure uncertainty, the textual prompt to a text-to-image model is compared to captions supplied by four image-to-text models (GIT, BLIP, BLIP-2, and InstructBLIP). Its evaluation is based on machine translation metrics (BLEU, ROUGE, METEOR, and SPICE) and word embedding’s cosine similarity (Word2Vec, GloVe, FastText, DistilRoBERTa, MiniLM-6, and MiniLM-12). The generative AI models performed consistently across the metrics; however, the vector space models yielded the highest average similarity, close to 80%, which suggests more ideal and “certain” results. Suggested future work includes identifying metrics that best align with a human baseline to ensure quality and consideration for more GAI models. The work within can be used to automatically identify concepts in which GAI is “uncertain” to drive research aimed at increasing confidence in these areas.

Document Retrieval System for Biomedical Question Answering

In this paper, we describe our biomedical document retrieval system and answers extraction module, which is part of the biomedical question answering system. Approximately 26.5 million PubMed articles are indexed as a corpus with the Apache Lucene text search engine. Our proposed system consists of three parts. The first part is the question analysis module, which analyzes the question and enriches it with biomedical concepts related to its wording. The second part of the system is the document retrieval module. In this step, the proposed system is tested using different information retrieval models, like the Vector Space Model, Okapi BM25, and Query Likelihood. The third part is the document re-ranking module, which is responsible for re-arranging the documents retrieved in the previous step. For this study, we tested our proposed system with 6B training questions from the BioASQ challenge task. We obtained the best MAP score on the document retrieval phase when we used Query Likelihood with the Dirichlet Smoothing model. We used the sequential dependence model at the re-rank phase, but this model produced a worse MAP score than the previous phase. In similarity calculation, we included the Named Entity Recognition (NER), UMLS Concept Unique Identifiers (CUI), and UMLS Semantic Types of the words in the question to find the sentences containing the answer. Using this approach, we observed a performance enhancement of roughly 25% for the top 20 outcomes, surpassing another method employed in this study, which relies solely on textual similarity.

Sentiment Analysis of Short Texts Using SVMs and VSMs-Based Multiclass Semantic Classification

ABSTRACT In our approach, a hybrid machine learning model is proposed which uses Enhanced Vector Space Model (EVSM) along with Hybrid Support Vector Machine (HSVM) classifier. Initially the social media-based information is retrieved using Enhanced Vector Space Model (EVSM). EVSMs are employed in order to characterize the text content by mapping them into high-dimensional vector spaces, capturing the relationships between words and their contextual meanings. Rigorous feature selection methods are employed to designate texts for review, and a multiclass semantic classification algorithm, specifically the HSVM classifier, is utilized for categorization. Decision tree algorithm is used along with SVM to refine the selection process. To enhance sentiment analysis accuracy, sentiment dictionaries are not only presented but also extended through the expansion of Stanford’s GloVE tool. To enhance precision, the proposed work introduces weight-enhancing methods for processing renowned text weights. Sentiments are classified into positive, negative, and neutral categories. Notably, the achieved results demonstrate improved accuracy, attributed to the incorporation of an emotional sentiment enhancement factor for determining weights and leveraging sentiment dictionaries for word availability. The accuracy is obtained to be 92.78% with 91.33% positive sentiment rate and 97.32% negative sentiment rate.

Semisupervised Learning-Based Word-Sense Disambiguation Using Word Embedding for Afaan Oromoo Language

Natural language is a type of language that human beings use to communicate with each other. However, it is very difficult to communicate with a machine-understandable language. Finding context meaning is challenging the activity of automatically identifying machine translation, indexing engines, and predicting neighbor words in natural language. Many researchers around the world investigated word-sense disambiguation in different languages, including the Afaan Oromo language, to solve this challenge. Nevertheless, the amount of effort for Afaan Oromo is very little in terms of finding context meaning and predicting neighbor words to solve the word ambiguity problem. Since the Afaan Oromo language is one of the languages developed in Ethiopia, it needs the latest technology to enhance communication and overcome ambiguity challenges. So far, this work aims to design and develop a vector space model for the Afaan Oromo language that can provide the application of word-sense disambiguation to increase the performance of information retrieval. In this work, the study has used the Afaan Oromo word embedding method to disambiguate a contextual meaning of words by applying the semisupervised technique. To conduct the study, 456,300 Afaan Oromo words were taken from different sources and preprocessed for experimentation by the Natural Language Toolkit and Anaconda tool. The K-means machine learning algorithm was used to cluster similar word vocabulary. Experimental results show that using word embedding for the proposed language’s corpus improves the performance of the system by a total accuracy of 98.89% and outperforms the existing similar systems.