Pre-trained Word Embedding Models Research Articles

LongT5Rank: A Novel Integrated Hybrid Approach for Text Summarisation

Text summarisation reduces text length while retaining important information, helping individuals, especially students, in managing information overload during research or assignments. However, existing text summarisation methods often lose important details, generate irrelevant or redundant sentences, or produce incoherent summaries. This study introduces a hybrid approach, LongT5Rank (coined in this study), which combines TextRank, an extractive summarisation algorithm, with LongT5, an abstractive summarisation algorithm, to automate the summarisation process. TextRank utilizes GloVe, a pre-trained word embedding model, and PageRank, a graph-based ranking algorithm, to select representative sentences. LongT5, an encoder-decoder transformer model, condenses extracted sentences into a concise and coherent summary, handling input sequences up to 16,384 tokens, for long-range sequence-to-sequence tasks. The LongT5Rank approach has shown significant achievements, including a minimum 60% compression rate, a minimum 0.6 semantic textual similarity score, and an improved F-measure compared to employing TextRank alone. Furthermore, it received positive feedback from Human Level Performance (HLP), underlining the importance of evaluating results directly from human users. This emphasizes the belief that the performance of the proposed solution should be assessed natively by humans. By combining both extractive and abstractive methods, LongT5Rank excels in generating accurate and coherent summaries.

Moral attitudes towards effort and efficiency: a comparison between American and Chinese history

In some cultures, merely exerting effort is considered virtuous, even when the effort is inefficient. Our study examines how this moral attitude towards effort (relative to efficiency) has evolved historically across two distinct sociopolitical and linguistic contexts: the People’s Republic of China and the United States, using natural language processing techniques. Specifically, two formal political corpora were used—the People’s Daily (1950–2021) and the Congressional speeches for the U.S. (1873–2011). We developed dictionaries for each concept based on pre-trained word embedding models in both languages. Moral attitudes towards effort and efficiency were calculated on a year-by-year basis as the cosine similarity between the dictionaries of these concepts and an existing dictionary of morality. We benchmarked the fluctuations of moral attitude towards inefficient effort against critical historical events in the two countries. Additional time series analysis and Granger tests revealed the association and potential directionality between the evolution of moral attitude towards inefficient effort and critical socio-cultural variables such as collectivism and cultural looseness. Our research sheds light on the historical and socio-cultural roots of moralization of effort and has implications for historical psychology research on moral attitudes.

Leveraging Pre-Trained Word Embedding Models for Fake Review Identification

Leveraging Pre-Trained Word Embedding Models for Fake Review Identification

A new word embedding model integrated with medical knowledge for deep learning-based sentiment classification

The development of intelligent systems that use social media data for decision-making processes in numerous domains such as politics, business, marketing, and finance, has been made possible by the popularity of social media platforms. However, the utilization of textual data from social media in the healthcare management industry is still somewhat limited when it is compared to other industries. Investigating how current machine learning and natural language processing technologies can be used in the healthcare industry to gauge public sentiment is an important study. Earlier works on healthcare sentiment analysis have utilized traditional word embedding models trained on the general and medical corpus. However, integration of medical knowledge to pre-trained word embedding models has not been considered yet. Word embedding models trained on the general corpus led to the problem of lacking medical knowledge and the models trained on the small size of the medical corpus have limitations in capturing semantic and syntactic properties. This research proposes a new word embedding model named Word Embedding Integrated with Medical Knowledge Vector (WE-iMKVec). The proposed model integrates sentiment lexicons and medical knowledgebases into the pre-trained word embedding to enrich the properties of word embedding. A new medical-aware sentiment polarity score is proposed for the utilization in learning neural-network sentiment and these vectors incorporate with the original pre-trained word vectors. The resulting vectors are enriched with lexicon vectors and the medical knowledge vectors: Adverse Drug Reaction (ADR) vector and Unified Medical Language System (UMLS) vector are used to build the proposed WE-iMKVec model. WE-iMKVec is validated on the five different social media healthcare review datasets and the empirical results showed its superiority over traditional word embedding models in medical sentiment analysis. The highest improvement can be found in the patients.info medical condition dataset where the proposed model outperforms three conventional word2vec models (Google-News, PubMed-PMC, and Drug Reviews) by 12.7 %, 31.4 %, and 25.4 % respectively in terms of F1 score.

Comparative Analysis of Long Short-Term Memory Architecture for Text Classification

Text classification which is a part of NLP is a grouping of objects in the form of text based on certain characteristics that show similarities between one document and another. One of methods used in text classification is LSTM. The performance of the LSTM method itself is influenced by several things such as datasets, architecture, and tools used to classify text. On this occasion, researchers analyse the effect of the number of layers in the LSTM architecture on the performance generated by the LSTM method. This research uses IMDB movie reviews data with a total of 50,000 data. The data consists of positive, negative data and there is data that does not yet have a label. IMDB Movie Reviews data go through several stages as follows: Data collection, data pre-processing, conversion to numerical format, text embedding using the pre-trained word embedding model: Fastext, train and test classification model using LSTM, finally validate and test the model so that the results are obtained from the stages of this research. The results of this study show that the one-layer LSTM architecture has the best accuracy compared to two-layer and three-layer LSTM with training accuracy and testing accuracy of one-layer LSTM which are 0.856 and 0.867. While the training accuracy and testing accuracy on two-layer LSTM are 0.846 and 0.854, the training accuracy and testing accuracy on three layers are 0.848 and 864.

Evaluating Familiarity Ratings of Domain Concepts with Interpretable Machine Learning: A Comparative Study

Psycholinguistic properties such as concept familiarity and concreteness have been investigated in relation to technological innovations in teaching and learning. Due to ongoing advances in semantic representation and machine learning technologies, the automatic extrapolation of lexical psycholinguistic properties has received increased attention across a number of disciplines in recent years. However, little attention has been paid to the reliable and interpretable assessment of familiarity ratings for domain concepts. To address this gap, we present a regression model grounded in advanced natural language processing and interpretable machine learning techniques that can predict domain concepts’ familiarity ratings based on their lexical features. Each domain concept is represented at both the orthographic–phonological level and semantic level by means of pretrained word embedding models. Then, we compare the performance of six tree-based regression models (adaptive boosting, gradient boosting, extreme gradient boosting, a light gradient boosting machine, categorical boosting, and a random forest) on domain concepts’ familiarity rating prediction. Experimental results show that categorical boosting with the lowest MAPE (0.09) and the highest R2 value (0.02) is best suited to predicting domain concepts’ familiarity. Experimental results also revealed the prospect of integrating tree-based regression models and interpretable machine learning techniques to expand psycholinguistic resources. Specifically, findings showed that the semantic information of raw words and parts of speech in domain concepts are reliable indicators when predicting familiarity ratings. Our study underlines the importance of leveraging domain concepts’ familiarity ratings; future research should aim to improve familiarity extrapolation methods. Scholars should also investigate the correlation between students’ engagement in online discussions and their familiarity with domain concepts.

Comparison of Pre-trained vs Custom-trained Word Embedding Models for Word Sense Disambiguation

The prime objective of word sense disambiguation (WSD) is to develop such machines that can automatically recognize the actual meaning (sense) of ambiguous words in a sentence. WSD can improve various NLP and HCI challenges. Researchers explored a wide variety of methods to resolve this issue of sense ambiguity. However, majorly, their focus was on English and some other well-reputed languages. Urdu with more than 300 million users and a large amount of electronic text available on the web is still unexplored. In recent years, for a variety of Natural Language Processing tasks, word embedding methods have proven extremely successful. This study evaluates, compares, and applies a variety of word embedding approaches to Urdu Word embedding (both Lexical Sample and All-Words), including pre-trained (Word2Vec, Glove, and FastText) as well as custom-trained (Word2Vec, Glove, and FastText trained on the Ur-Mono corpus). Two benchmark corpora are used for the evaluation in this study: (1) the UAW-WSD-18 corpus and (2) the ULS-WSD-18 corpus. For Urdu All-Words WSD tasks, top results have been achieved (Accuracy=60.07 and F1=0.45) using pre-trained FastText. For the Lexical Sample, WSD has been achieved (Accuracy=70.93 and F1=0.60) using custom-trained GloVe word embedding method.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

DISINFORMATION DETECTION ABOUT ISLAMIC ISSUES ON SOCIAL MEDIA USING DEEP LEARNING TECHNIQUES

Nowadays, many people receive news and information about what is happening around them from social media networks. These social media platforms are available free of charge and allow anyone to post news or information or express their opinion without any restrictions or verification, thus contributing to the dissemination of disinformation. Recently, disinformation about Islam has spread through pages and groups on social media dedicated to attacking the Islamic religion. Many studies have provided models for detecting fake news or misleading information in many domains, such as political, social, economic, and medical, except in the Islamic domain. Due to this negative impact of spreading disinformation targeting the Islamic religion, there is an increase in Islamophobia, which threatens societal peace. In this paper, we present a Bidirectional Long Short-Term Memory-based model trained on an Islamic dataset (RIDI) that was collected and labeled by two separate specialized groups. In addition, using a pre-trained word-embedding model will generate Out-Of-Vocabulary, because it deals with a specific domain. To address this issue, we have retrained the pre-trained Glove model on Islamic documents using the Mittens method. The results of the experiments proved that our proposed model based on Bidirectional Long Short-Term Memory with the retrained Glove model on the Islamic articles is efficient in dealing with text sequences better than unidirectional models and provides a detection accuracy of 95.42% of Area under the ROC Curve measure compared to the other models.

Improved Arabic image captioning model using feature concatenation with pre-trained word embedding

Automatic captioning of images contributes to identifying features of multimedia content and helps in the detection of interesting patterns, trends, and occurrences. English image captioning has recently made incredible progress; however, Arabic image captioning is still lagging. In the field of machine learning, Arabic image-caption generation is generally a very difficult problem. This paper presents a more accurate model for Arabic image captioning by using transformer models in both the encoder and decoder phases as feature extractors from images in the encoder phase and a pre-trained word embedding model in the decoder phase. The models are demonstrated, and all of them are implemented, trained, and tested on Arabic Flickr8k datasets. For the image feature extraction subsystem, we compared using three different individual vision models (SWIN, XCIT, and ConvNexT) with concatenation to get among them the most expressive extracted feature vector of the image, and for the caption generation lingual subsystem, which is tested by four different pre-trained language embedding models: (ARABERT, ARAELECTRA, MARBERTv2, and CamelBERT), to select from them the most accurate pre-trained language embedding model. Our experiments showed that building an Arabic image captioning system that uses a concatenation of the three transformer-based models ConvNexT combined with SWIN and XCIT as an image feature extractor, combined with the CamelBERT language embedding model produces the best results among the other combinations, having scores of 0.5980 with BLEU-1 and with ConvNexT combined with SWIN the araelectra language embedding model having a score of 0.1664 with BLEU-4 which are higher than the previously reported values of 0.443 and 0.157.

Evaluation of effectiveness in conversations between humans and chatbots using parallel convolutional neural networks with multiple temporal resolutions

Chatbots enable the automation of several components in customer service and allow the support of multiple users. Despite their multiple advantages, due to the large amount of conversations generated by a chatbot, it is difficult to determine whether customer requests are well-addressed. For practical reasons, chatbot’s effectiveness is evaluated manually based upon a small sample (randomly chosen) of conversations or through self-reported user satisfaction. This procedure does not guarantee the correct evaluation of the service because the sample is generally not large enough and self-reports might be influenced by different external factors not directly associated to the chatbot’s functioning. This study proposes a methodology for automatic evaluation of chatbot effectiveness in real production environments. The analysis considers convolutional neural networks adapted for natural language processing, using two parallel convolutional layers to evaluate questions and answers independently. The proposed model also incorporates filters to extract features with multiple temporal resolution. This methodology is tested upon real conversations of chatbots that provide service to two different companies. The results are compared to baseline models based on classical techniques with different pre-trained word embedding models. According to our results, the proposed approach provides accuracies between 78.95% and 80.18%, which outperforms the best result of the baseline models by 2.9%.

Developing an Incivility Dictionary for German Online Discussions – a Semi-Automated Approach Combining Human and Artificial Knowledge

ABSTRACT Incivility in online discussions has become an important issue in political communication research. Instruments and tools for the automated analysis of uncivil content, however, are rare, especially for non-English user-generated text. In this study, we present a) an extensive dictionary (DIKI - Diktionär für Inzivilität, English: Dictionary for Incivility) to detect incivility in German-language online discussions, and b) a semi-automated two-step-approach that combines manual content analysis with automated keyword collection using a pre-trained word embedding model. We show that DIKI clearly outperforms comparable dictionaries that have been used as alternative instruments to measure incivility (e.g., the LIWC) as well as basic machine learning approaches to text classification. Further, we provide evidence that pre-trained word embeddings can fruitfully be employed in the explorative phase of creating dictionaries. Still, the manual evaluation of DIKI confirms that detecting complex and context-dependent forms of incivility remains challenging and constant update would be needed to maintain performance. Finally, the detailed documentation of the developing and evaluation process of DIKI may serve as a guideline for further research. We therefore provide DIKI as a freely available instrument that also will be applicable in a web interface for drag-and-drop data analysis (diki.limitedminds.org).

A reproducible experimental survey on biomedical sentence similarity: A string-based method sets the state of the art.

This registered report introduces the largest, and for the first time, reproducible experimental survey on biomedical sentence similarity with the following aims: (1) to elucidate the state of the art of the problem; (2) to solve some reproducibility problems preventing the evaluation of most current methods; (3) to evaluate several unexplored sentence similarity methods; (4) to evaluate for the first time an unexplored benchmark, called Corpus-Transcriptional-Regulation (CTR); (5) to carry out a study on the impact of the pre-processing stages and Named Entity Recognition (NER) tools on the performance of the sentence similarity methods; and finally, (6) to bridge the lack of software and data reproducibility resources for methods and experiments in this line of research. Our reproducible experimental survey is based on a single software platform, which is provided with a detailed reproducibility protocol and dataset as supplementary material to allow the exact replication of all our experiments and results. In addition, we introduce a new aggregated string-based sentence similarity method, called LiBlock, together with eight variants of current ontology-based methods, and a new pre-trained word embedding model trained on the full-text articles in the PMC-BioC corpus. Our experiments show that our novel string-based measure establishes the new state of the art in sentence similarity analysis in the biomedical domain and significantly outperforms all the methods evaluated herein, with the only exception of one ontology-based method. Likewise, our experiments confirm that the pre-processing stages, and the choice of the NER tool for ontology-based methods, have a very significant impact on the performance of the sentence similarity methods. We also detail some drawbacks and limitations of current methods, and highlight the need to refine the current benchmarks. Finally, a notable finding is that our new string-based method significantly outperforms all state-of-the-art Machine Learning (ML) models evaluated herein.

Extractive Summarization on Food Reviews

Text summarization is a technique to summarize the content of a sizeable text but meanwhile it keep the key information. Extractive summarization and abstractive summarization are the main techniques for text summarization. TextRank algorithm, an extractive summarization technique is applied to perform automatic text summarization in this study. Furthermore, GloVe pre-trained word embedding model is used to map each word from the reviews to vector representation. In the end, PageRank algorithm is applied to rank the sentences based on their sentence ranking scores. The more important and relevant sentences which can be the representatives of summary will be placed in a higher rank. The objective of our study is to extract top five reviews with the highest sentence ranking scores which can form a summary to provide a conspectus of a cookies brand in Amazon food reviews. An analysis of the customer perception based on the summary generated is conducted to understand their needs and level of satisfactions. The final summary demonstrates that Amazon customer reviews for certain cookies brand are generally positive.

Learning High-Order Semantic Representation for Intent Classification and Slot Filling on Low-Resource Language via Hypergraph

Representation of language is the first and critical task for Natural Language Understanding (NLU) in a dialogue system. Pretraining, embedding model, and fine-tuning for intent classification and slot-filling are popular and well-performing approaches but are time consuming and inefficient for low-resource languages. Concretely, the out-of-vocabulary and transferring to different languages are two tough challenges for multilingual pretrained and cross-lingual transferring models. Furthermore, quality-proved parallel data are necessary for the current frameworks. Stepping over these challenges, different from the existing solutions, we propose a novel approach, the Hypergraph Transfer Encoding Network “HGTransEnNet. The proposed model leverages off-the-shelf high-quality pretrained word embedding models of resource-rich languages to learn the high-order semantic representation of low-resource languages in a transductive clustering manner of hypergraph modeling, which does not need parallel data. The experiments show that the representations learned by “HGTransEnNet” for low-resource language are more effective than the state-of-the-art language models, which are pretrained on a large-scale multilingual or monolingual corpus, in intent classification and slot-filling tasks on Indonesian and English datasets.

Auto Labeling Methods Developed Through Semi-Weakly Supervised Learning in Prognostics and Health Management Applications for Rolling Ball Bearing

In recent years, complex machine learning and deep learning algorithms have been used in prognostics and health management applications to monitor the state of industrial systems in real-time and predict failures in advance. However, the performance of these algorithms depends on the size of the collected labeled datasets. This study proposes a semi-weakly supervised learning method that creates label functions using a small amount of data and, consequently, combines weak supervision and semi-supervision methods to expand the labeled dataset for training. The proposed model training method creates labels automatically in the life prediction dataset of rolling element bearings, which were manually labeled by expert groups. The performance of the deep neural network is continuously improved through experiments, and a performance improvement similar to that of manually labeled training data is presented. Future research applications include meta-learner experiments (e.g., use of pre-trained word embedding or other model architectures) and more appropriate selection methods and definitions for the basic learner.

Extracting Prominent Aspects of Online Customer Reviews: A Data-Driven Approach to Big Data Analytics

Sentiment analysis on social media and e-markets has become an emerging trend. Extracting aspect terms for structure-free text is the primary task incorporated in the aspect-based sentiment analysis. This significance relies on the dependency of other tasks on the results it provides, which directly influences the accuracy of the final results of the sentiment analysis. In this work, we propose an aspect term extraction model to identify the prominent aspects. The model is based on clustering the word vectors generated using the pre-trained word embedding model. Dimensionality reduction was employed to improve the quality of word clusters obtained using the K-Means++ clustering algorithm. The proposed model was tested on the real datasets collected from online retailers’ websites and the SemEval-14 dataset. Results show that our model outperforms the baseline models.

SMAD: SMart assistant during and after a medical emergency case based on deep learning sentiment analysis: The pandemic COVID-19 case.

The huge cost of emergency situations could have fatal effects on humanity and society, and it could present a genuine threat to both of them. In fact, most people confronted with an emergency could feel psychological trauma, which will, for the most part, change over time as they can exhibit chaotic or even turbulent behaviours. The situation could worsen in the case of a pandemic as fear and anxiety invade and spread in addition to isolation and quarantine. In this paper, we propose to build a smart assistant, called SMAD, that could detect the symptoms of an emergency case as well as symptoms of a mental disorder while analysing the natural language speech of an ordinary citizen, during and after an emergency situation using natural language processing and deep learning sentiment analysis model to track the patient’s mental state during an ongoing conversation. Our proposed smart assistant is an online human-bot interaction that could handle a variety of physical and mental circumstances of any emergency situation. The proposed approach is a smart healthcare service that consists of four interconnected modules: The information understanding module, the data collector module, the action generator module, and the mental analysis module, which is based on the sentiment analysis model performed on a social media dataset using a pre-trained word-embedding model.

Understanding what patients think about hospitals: A deep learning approach for detecting emotions in patient opinions

IntroductionMost hospital assessment systems are based on the study of objective statistical variables as well as patient opinions on their experiences with respect to the services offered by each hospital. Nevertheless, studies have indicated that most of these assessment systems fail to detect patient emotions when they are assessing their stays in a hospital. This information is vital to understanding most of the patient reviews, which are very complex and convey several emotions per review. Therefore, this study aimed to address the problem of detecting multiple emotions from patient reviews. MethodsFirst, a large set of patient opinions was collected from a website that allowed patients to publish their experiences when visiting hospitals. Second, each opinion was labeled with the corresponding conveyed emotions. Third, a deep learning architecture based on a bidirectional gated recurrent unit with a multichannel convolutional neural network layer was proposed to detect multiple emotions from these reviews. Finally, the hyperparameters of this architecture were fine-tuned and different pretrained word embedding models were configured to test its performance. ResultsThe results confirmed that our proposed method outperformed other deep learning and machine learning-based algorithms and achieved an average accuracy of 95.82%. Furthermore, the experiments show that clinical-domain word embedding slightly outperforms other general-domain word embeddings, although general-domain embeddings are larger in terms of dimensions. ConclusionsThe combination of the gated recurrent unit and the multichannel convolutional neural network is able to exploit both semantic and syntactic characteristics of patient opinions. The findings of this study identify research gaps related to areas such as opinion-based hospital recommendations, thereby providing future research directions.

Sentiment analysis on Bangla text using extended lexicon dictionary and deep learning algorithms

Sentiment analysis (SA) is a subset of natural language processing (NLP) research. In the case of categorical weighted based dictionary with rule-based sentiment score generation, no work in SA has been done yet in Bangla language using deep learning (DL) approaches. This paper proposes DL models for SA on Bangla text using an extended lexicon data dictionary (LDD). We implement the rule-based method Bangla text sentiment score (BTSC) algorithm for extracting polarity from large texts. These polarities are then fed into the neural network along with the preprocessed text as training samples. The preprocessed texts are formatted as a vectorization of words of unique numbers of pre-trained word embedding models. Word2Vec matrix with top highest probability word is applied on embedding layer as a weighted matrix to fit the DL models. This paper also presents a remarkably detailed analysis of selective DL models with some fine tuning. The fine-tuning includes the use of drop out, optimizer regularization, learning rate, multiple layers, filters, attention mechanism, capsule layers, transformer with progressive training along with validation and testing accuracy, precision, recall and F1-score. Experimental results indicate that the proposed new long short-term memory (LSTM) models are highly accurate in performing SA tasks. For our proposed hierarchical attention based LSTM (HAN-LSTM), Dynamic routing based capsule neural network with Bi-LSTM (D-CAPSNET-Bi-LSTM) and bidirectional encoder representations from Transformers (BERT) with LSTM (BERT-LSTM) model we achieved accuracy values of 78.52%, 80.82% and 84.18% respectively.