Word Embedding Representation Research Articles

Automatic Topic Title Assignment with Word Embedding

AbstractIn this paper, we propose TAWE (title assignment with word embedding), a new method to automatically assign titles to topics inferred from sets of documents. This method combines the results obtained from the topic modeling performed with, e.g., latent Dirichlet allocation (LDA) or other suitable methods and the word embedding representation of words in a vector space. This representation preserves the meaning of the words while allowing to find the most suitable word that represents the topic. The procedure is twofold: first, a cleaned text is used to build the LDA model to infer a desirable number of latent topics; second, a reasonable number of words and their weights are extracted from each topic and represented in n-dimensional space using word embedding. Based on the selected weighted words, a centroid is computed, and the closest word is chosen as the title of the topic. To test the method, we used a collection of tweets about climate change downloaded from some of the main newspapers accounts on Twitter. Results showed that TAWE is a suitable method for automatically assigning a topic title.

Pipelining Semantic Expansion and Noise Filtering for Sentiment Analysis of Short Documents – CluSent Method

The challenge of constructing effective sentiment models is exacerbated by a lack of sufficient information, particularly in short texts. Enhancing short texts with semantic relationships becomes crucial for capturing affective nuances and improving model efficacy, albeit with the potential drawback of introducing noise. This article introduces a novel approach, CluSent, designed for customized dataset-oriented sentiment analysis. CluSent capitalizes on the CluWords concept, a proposed powerful representation of semantically related words. To address the issues of information scarcity and noise, CluSent addresses these challenges: (i) leveraging the semantic neighborhood of pre-trained word embedding representations to enrich document representation and (ii) introducing dataset-specific filtering and weighting mechanisms to manage noise. These mechanisms utilize part-of-speech and polarity/intensity information from lexicons. In an extensive experimental evaluation spanning 19 datasets and five state-of-the-art baselines, including modern transformer architectures, CluSent emerged as the superior method in the majority of scenarios (28 out of 38 possibilities), demonstrating noteworthy performance gains of up to 14% over the strongest baselines.

MulTFBS: A Spatial-Temporal Network with Multichannels for Predicting Transcription Factor Binding Sites.

Revealing the mechanisms that influence transcription factor binding specificity is the key to understanding gene regulation. In previous studies, DNA double helix structure and one-hot embedding have been used successfully to design computational methods for predicting transcription factor binding sites (TFBSs). However, DNA sequence as a kind of biological language, the method of word embedding representation in natural language processing, has not been considered properly in TFBS prediction models. In our work, we integrate different types of features of DNA sequence to design a multichanneled deep learning framework, namely MulTFBS, in which independent one-hot encoding, word embedding encoding, which can incorporate contextual information and extract the global features of the sequences, and double helix three-dimensional structural features have been trained in different channels. To extract sequence high-level information effectively, in our deep learning framework, we select the spatial-temporal network by combining convolutional neural networks and bidirectional long short-term memory networks with attention mechanism. Compared with six state-of-the-art methods on 66 universal protein-binding microarray data sets of different transcription factors, MulTFBS performs best on all data sets in the regression tasks, with the average R2 of 0.698 and the average PCC of 0.833, which are 5.4% and 3.2% higher, respectively, than the suboptimal method CRPTS. In addition, we evaluate the classification performance of MulTFBS for distinguishing bound or unbound regions on TF ChIP-seq data. The results show that our framework also performs well in the TFBS classification tasks.

Leveraging Large Language Models for Sensor Data Retrieval

The growing significance of sensor data in the development of information technology services finds obstacles due to disparate data presentations and non-adherence to FAIR principles. This paper introduces a novel approach for sensor data gathering and retrieval. The proposal leverages large language models to convert sensor data into FAIR-compliant formats and to provide word embedding representations of tabular data for subsequent exploration, enabling semantic comparison. The proposed system comprises two primary components. The first focuses on gathering data from sensors and converting it into a reusable structured format, while the second component aims to identify the most relevant sensor data to augment a given user-provided dataset. The evaluation of the proposed approach involved comparing the performance of various large language models in generating representative word embeddings for each table to retrieve related sensor data. The results show promising performance in terms of precision and MRR (0.90 and 0.94 for the best-performing model, respectively), indicating the system’s ability to retrieve pertinent sensor data that fulfil user requirements.

Assessing the Impact of Static, Contextual and Character Embeddings for Arabic Machine Translation

Word embeddings/representations are an important component of Natural Language Processing (NLP) tasks. Most Neural Machine Translation (NMT) systems that use such representations disregard word morphology by assigning a unique vector to each unique word in the used vocabulary and thus cannot handle Out-Of-Vocabulary (OOV) words. In some languages, such as Arabic, the meaning of words is associated with the meaning of the individual characters that constitute them, as these characters embody internal information. In this study, a combination of character- and word- level models is used to determine the most effective approaches to semantically and morphologically representing affective Arabic words. Furthermore, this work examines the strategy of combining static, character and contextual word embeddings to obtain richer representations for the Arabic Machine Translation (MT) task. To the best of our knowledge, we are the first to investigate the combination of static word embeddings, contextual ones and character-level representation in Arabic MT. Furthermore, a Deep Learning (DL) architecture is employed on data preprocessed by various prominent preprocessing techniques. Various experiments were conducted and the findings indicate that the integration of various models for word embedding and character-level representation is feasible and more effective than the state-of-the-art Arabic MT systems.

Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis

Human Multimodal Sentiment Analysis (MSA) is an attractive research that studies sentiment expressed from multiple heterogeneous modalities. While transformer-based methods have achieved great success, designing an effective “co-attention” model to associate text modality with nonverbal modalities remains challenging. There are two main problems: 1) the dominant role of the text in modalities is underutilization, and 2) the interaction between modalities is not sufficiently explored. This paper proposes a deep modular Co-Attention Shifting Network (CoASN) for MSA. A Cross-modal Modulation Module based on Co-attention (CMMC) and an Advanced Modality-mixing Adaptation Gate (AMAG) are constructed. The CMMC consists of the Text-guided Co-Attention (TCA) and Interior Transformer Encoder (ITE) units to capture inter-modal features and intra-modal features. With text modality as the core, the CMMC module aims to guide and promote the expression of emotion in nonverbal modalities, and the nonverbal modalities increase the richness of the text-based multimodal sentiment information. In addition, the AMAG module is introduced to explore the dynamical correlations among all modalities. Particularly, this efficient module first captures the nonverbal shifted representations and then combines them to calculate the shifted word embedding representations for the final MSA tasks. Extensive experiments on two commonly used datasets, CMU-MOSI and CMU-MOSEI, demonstrate that our proposed method is superior to the state-of-the-art performance.

Research on English Reading Comprehension Strategies Based on Natural Language Processing

Abstract This study investigates strategies based on natural language processing to improve the efficiency and effectiveness of English reading comprehension by analyzing and optimizing reading comprehension models. We introduce the main framework of the English reading comprehension task, including the word embedding representation layer, the context encoding layer, the information interaction layer, and the output layer. The article utilizes the Word2vec model and decision tree ID3 algorithm to enhance the efficiency and accuracy of reading comprehension. The research data comprised 800 English articles, with 60% used for training and 40% used for testing. The articles were graded for difficulty and resources were pushed. It was found that the model proposed in this study performs better in difficulty grading (with a rating range of 0.797-0.926) and is more efficient in terms of time overhead (feature extraction time of about 10 milliseconds, and grading time of 1.7-2.6 seconds), compared to the traditional models such as BiLSTM and RNN. The pushed resources significantly correlate with learners’ self-assessed difficulty and actual performance. The NLP-based strategy proposed in the paper shows significant advantages in both efficiency and accuracy, and helps to improve English learners’ reading comprehension.

Research on the mechanism of teaching culture in Civics course based on deep learning model

Abstract This paper uses convolutional neural networks to construct a deep learning-based teaching mechanism for question-and-answer on civic culture. The one-hot representation and the distributed representation of words in the distributed expression form of natural language are analyzed. Word2Vec, a word embedding training tool, is used to generate word embedding representations, and CNN is applied to store and classify word semantics so as to generate semantic matrices. The attention mechanism is combined to simplify the model processing steps and reduce the complexity of operations. The deep learning hierarchical semantic extraction model is tested by loss function and relative similarity to judge the accuracy and performance of the model’s operation. The results showed that the mean composite score of the experimental group was 5.2 points higher than that of the control group, and the t-test results P=0.003<0.05 indicated that there was a significant difference between the overall assessment scores of the two groups, and the teaching of Civic Culture Quiz was beneficial to the improvement of students’ performance.

Animal disease surveillance: How to represent textual data for classifying epidemiological information.

The value of informal sources in increasing the timeliness of disease outbreak detection and providing detailed epidemiological information in the early warning and preparedness context is recognized. This study evaluates machine learning methods for classifying information from animal disease-related news at a fine-grained level (i.e., epidemiological topic). We compare two textual representations, the bag-of-words method and a distributional approach, i.e., word embeddings. Both representations performed well for binary relevance classification (F-measure of 0.839 and 0.871, respectively). Bag-of-words representation was outperformed by word embedding representation for classifying sentences into fine-grained epidemiological topics (F-measure of 0.745). Our results suggest that the word embedding approach is of interest in the context of low-frequency classes in a specialized domain. However, this representation did not bring significant performance improvements for binary relevance classification, indicating that the textual representation should be adapted to each classification task.

An Innovative Arabic Word Embedding Representation for Enhanced Sentiment Analysis

An Innovative Arabic Word Embedding Representation for Enhanced Sentiment Analysis

Advances in monolingual and crosslingual automatic disability annotation in Spanish

BackgroundUnlike diseases, automatic recognition of disabilities has not received the same attention in the area of medical NLP. Progress in this direction is hampered by obstacles like the lack of annotated corpus. Neural architectures learn to translate sequences from spontaneous representations into their corresponding standard representations given a set of samples. The aim of this paper is to present the last advances in monolingual (Spanish) and crosslingual (from English to Spanish and vice versa) automatic disability annotation. The task consists of identifying disability mentions in medical texts written in Spanish within a collection of abstracts from journal papers related to the biomedical domain.ResultsIn order to carry out the task, we have combined deep learning models that use different embedding granularities for sequence to sequence tagging with a simple acronym and abbreviation detection module to boost the coverage.ConclusionsOur monolingual experiments demonstrate that a good combination of different word embedding representations provide better results than single representations, significantly outperforming the state of the art in disability annotation in Spanish. Additionally, we have experimented crosslingual transfer (zero-shot) for disability annotation between English and Spanish with interesting results that might help overcoming the data scarcity bottleneck, specially significant for the disabilities.

Towards Safe Cyber Practices: Developing a Proactive Cyber-Threat Intelligence System for Dark Web Forum Content by Identifying Cybercrimes

The untraceable part of the Deep Web, also known as the Dark Web, is one of the most used “secretive spaces” to execute all sorts of illegal and criminal activities by terrorists, cybercriminals, spies, and offenders. Identifying actions, products, and offenders on the Dark Web is challenging due to its size, intractability, and anonymity. Therefore, it is crucial to intelligently enforce tools and techniques capable of identifying the activities of the Dark Web to assist law enforcement agencies as a support system. Therefore, this study proposes four deep learning architectures (RNN, CNN, LSTM, and Transformer)-based classification models using the pre-trained word embedding representations to identify illicit activities related to cybercrimes on Dark Web forums. We used the Agora dataset derived from the DarkNet market archive, which lists 109 activities by category. The listings in the dataset are vaguely described, and several data points are untagged, which rules out the automatic labeling of category items as target classes. Hence, to overcome this constraint, we applied a meticulously designed human annotation scheme to annotate the data, taking into account all the attributes to infer the context. In this research, we conducted comprehensive evaluations to assess the performance of our proposed approach. Our proposed BERT-based classification model achieved an accuracy score of 96%. Given the unbalancedness of the experimental data, our results indicate the advantage of our tailored data preprocessing strategies and validate our annotation scheme. Thus, in real-world scenarios, our work can be used to analyze Dark Web forums and identify cybercrimes by law enforcement agencies and can pave the path to develop sophisticated systems as per the requirements.

Improving BERT model for requirements classification by bidirectional LSTM-CNN deep model

In the last decade, requirements classification has emerged as hot research topic in Requirements Engineering (RE). Early identification of software requirements helps the development team in the design of software systems. Manual identification and classification of these requirements is time-consuming and labor-intensive task. To address this issue, machine and deep learning techniques have been studied for automatic classification of requirements. Furthermore, an efficient word embedding representation of the input data is a major concern in automatic approaches. This research work presents a novel requirements classification model that integrates Bidirectional Encoder Representations from Transformers (BERT) with Bidirectional Long-short Term Memory (BiLSTM) with Convolutional Neural Network (CNN) layer called BERT-BiCNN. In BERT-BiCNN, BERT is used as word embedding layer that extract the full combination of contextual semantics. BiLSTM obtains contextual information. The CNN mechanism has been employed to reduce the dimensionality of feature space by selecting the important features. The effectiveness of BERT-BiCNN is evaluated on the PROMISE dataset. On comparative analysis, it can be concluded that proposed approach outperforms six recent deep learning based architectures in binary and multi-class classification.

Towards Enhanced Identification of Emotion from Resource-Constrained Language through a novel Multilingual BERT Approach

Emotion identification from text has recently gained attention due to its versatile ability to analyze human-machine interaction. This work focuses on detecting emotions from textual data. Languages, like English, Chinese, and German are widely used for text classification, however, limited research is done on resource-poor oriental languages. Roman Urdu (RU) is a resource-constrained language extensively used across Asia. This work focuses on predicting emotions from RU text. For this, a dataset is collected from different social media domains and based on Paul Ekman's theory it is annotated with six basic emotions, i.e., happy, surprise, angry, sad, fear, and disgusting. Dense word embedding representations of different languages is adopted that utilize existing pre-trained models. BERT is additionally pre-trained and fine-tuned for the classification task. The proposed approach is compared with baseline machine learning and deep learning algorithms. Additionally, a comparison of the current work is also performed with different approaches for the same task. Based on the empirical evaluation, the proposed approach performs better than the existing state-of-the-art with an average accuracy of 91%.

Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space.

Word embedding representations have been shown to be effective in predicting human neural responses to lingual stimuli. While these representations are sensitive to the textual context, they lack the extratextual sources of context such as prior knowledge, thoughts, and beliefs, all of which constitute the listener's perspective. In this study, we propose conceptualizing the listeners' perspective as a source that induces changes in the embedding space. We relied on functional magnetic resonance imaging data collected by Yeshurun Y, Swanson S, Simony E, Chen J, Lazaridi C, Honey CJ, Hasson U. Same story, different story: the neural representation of interpretive frameworks. Psychol Sci. 2017:28(3):307-319, in which two groups of human listeners (n= 40) were listening to the same story but with different perspectives. Using a dedicated fine-tuning process, we created two modified versions of a word embedding space, corresponding to the two groups of listeners. We found that each transformed space was better fitted with neural responses of the corresponding group, and that the spatial distances between these spaces reflect both interpretational differences between the perspectives and the group-level neural differences. Together, our results demonstrate how aligning a continuous embedding space to a specific context can provide a novel way of modeling listeners' intrinsic perspectives.

RepSum: A general abstractive summarization framework with dynamic word embedding representation correction

Abstractive summarization is flexible and allows the model to generate new words and phrases. However, the familiar words are more likely to be selected as abstract candidate words in the process of abstractive summarization, causing the generated abstract to diverge from the reference. In our consideration, this is caused by representation degeneration of the pre-trained word embedding. Therefore, this paper proposes a general abstractive summarization framework with dynamic word embedding representation correction (RepSum). The representation correction algorithm identifies the dimension most relevant to word frequency and eliminates the word frequency features. Then the distribution of word embeddings will be more even. As a result, the words will be selected as candidate words without frequency bias to improve the quality of the abstract. The experimental results illustrate that RepSum performs better than the benchmark model in summary quality, demonstrating our method’s effectiveness.

Improved Text Matching Model Based on BERT

Text matching is a basic and important task in natural language understanding, this paper proposes a new model BBMC for the problem of insufficient feature extraction ability of existing text matching models, which integrates BiLSTM and multi-scale CNN on the basis of BERT. First, the word embedding representation of the text is obtained by the BERT, and then the semantic features of the text are further extracted by the double-layer BiLSTM, followed by the multi-scale CNN model, the key local features are extracted, and finally the linear and SoftMax function are used to classify. Experimental results on the LCQMC dataset show that the BBMC has been improved to a certain extent compared with other methods, and the accuracy on the test set can be best achieved 88.01%.

A machine learning approach to recognize bias and discrimination in job advertisements

In recent years, the work of organizations in the area of digitization has intensified significantly. This trend is also evident in the field of recruitment where job application tracking systems (ATS) have been developed to allow job advertisements to be published online. However, recent studies have shown that recruiting in most organizations is not inclusive, being subject to human biases and prejudices. Most discrimination activities appear early but subtly in the hiring process, for instance, exclusive phrasing in job advertisement discourages qualified applicants from minority groups from applying. The existing works are limited to analyzing, categorizing and highlighting the occurrence of bias in the recruitment process. In this paper, we go beyond this and develop machine learning models for identifying and classifying biased and discriminatory language in job descriptions. We develop and evaluate a machine learning system for identifying five major categories of biased and discriminatory language in job advertisements, i.e., masculine-coded, feminine-coded, exclusive, LGBTQ-coded, demographic and racial language. We utilized the combination of linguistic features with recent state-of-the-art word embeddings representations as input features for various machine learning classifiers. Our results show that the machine learning classifiers were able to identify all the five categories of biased and discriminatory language with a decent accuracy. The Random Forest classifier with FastText word embeddings achieved the best performance with tenfolds cross-validation. Our system directly addresses the bias in the attraction phase of hiring by identifying and classifying biased and discriminatory language and thus encouraging recruiters to write more inclusive job advertisements.

Application of edge computing combined with deep learning model in the dynamic evolution of network public opinion in emergencies.

The aim is to clarify the evolution mechanism of Network Public Opinion (NPO) in public emergencies. This work makes up for the insufficient semantic understanding in NPO-oriented emotion analysis and tries to maintain social harmony and stability. The combination of the Edge Computing (EC) and Deep Learning (DL) model is applied to the NPO-oriented Emotion Recognition Model (ERM). Firstly, the NPO on public emergencies is introduced. Secondly, three types of NPO emergencies are selected as research cases. An emotional rule system is established based on the One-Class Classification (OCC) model as emotional standards. The word embedding representation method represents the preprocessed Weibo text data. Convolutional Neural Network (CNN) is used as the classifier. The NPO-oriented ERM is implemented on CNN and verified through comparative experiments after the CNN's hyperparameters are adjusted. The research results show that the text annotation of the NPO based on OCC emotion rules can obtain better recognition performance. Additionally, the recognition effect of the improved CNN is significantly higher than the Support Vector Machine (SVM) in traditional Machine Learning (ML). This work realizes the technological innovation of automatic emotion recognition of NPO groups and provides a basis for the relevant government agencies to handle the NPO in public emergencies scientifically.

Gender Prediction of Indonesian Twitter Users Using Tweet and Profile Features

The increasing use of social media generates huge amounts of data which in turn triggers research into social media analytics. Social media contents can be analyzed to explore public opinion on an issue or provide the insights reflecting proxy indicators towards real-world events. Understanding the demographics of social media users can increase the potential for applications of sentiment analysis, topic modeling, and other analytical tasks. To map demographics, we need to know the latent attributes of users, such as age, gender, occupation and location of residence. Since this attribute is not directly available, we need to do some inference from the social media data. This study aims to predict the gender attribute given a Twitter user account. We conducted experiments with several supervised classifiers with feature extraction, including the use of word embedding representations. The results of this study indicate that the combination of features extracted from Tweet contents and user profile structured data can predict the gender of Twitter users in Indonesia with accuracy above 80%.