Semantic Representations Of Words Research Articles

Sentiment analysis method of consumer comment text based on BERT and hierarchical attention in e-commerce big data environment

Abstract This study proposes an emotional analysis method of consumer comment text based on Bidirectional Encoder Representations from Transformers (BERT) and hierarchical attention. First, using the BERT pre-training model, the left and right contextual information is fused to enhance the semantic representation of words and generate dynamic word vectors containing contextual semantics. Second, the bidirectional long short-term memory network is used to obtain the sequence feature matrix, and the sentence representation and the text representation are obtained using the two-layer long short-term memory. Finally, the local attention mechanism and the global attention mechanism are introduced into the sentence representation layer and the text representation layer, respectively, and the text emotion of consumer comments is classified by softmax. Experiments show that the accuracy of the proposed method in Laptop data set is 93.01% and that in Restaurant data set is 92.45%. Therefore, the performance of the proposed method in the emotional analysis of consumer comment text is significantly better than that of the comparison method.

Improved conversational recommender system based on dialog context

Abstract Conversational recommender system (CRS) needs to be seamlessly integrated between the two modules of recommendation and dialog, aiming to recommend high-quality items to users through multiple rounds of interactive dialogs. Items can typically refer to goods, movies, news, etc. Through this form of interactive dialog, users can express their preferences in real time, and the system can fully understand the user’s thoughts and recommend corresponding items. Although mainstream dialog recommendation systems have improved the performance to some extent, there are still some key issues, such as insufficient consideration of the entity’s order in the dialog, the different contributions of items in the dialog history, and the low diversity of generated responses. To address these shortcomings, we propose an improved dialog context model based on time-series features. Firstly, we augment the semantic representation of words and items using two external knowledge graphs and align the semantic space using mutual information maximization techniques. Secondly, we add a retrieval model to the dialog recommendation system to provide auxiliary information for generating replies. We then utilize a deep timing network to serialize the dialog content and more accurately learn the feature relationship between users and items for recommendation. In this paper, the dialog recommendation system is divided into two components, and different evaluation indicators are used to evaluate the performance of the dialog component and the recommendation component. Experimental results on widely used benchmarks show that the proposed method is effective.

Effects of word emotional experience and participant emotionality in lexical decision

BackgroundPrevious research has shown that the emotional content of words affects how quickly they are recognised. One recent measure of word emotionality is emotional experience that measures the degree to which reading a word can invoke emotional experiences tied to the word. Words that are higher in emotional experience are recognised more rapidly in the lexical decision task.MethodsThe current study examines how words' emotional experience and participants' emotionality affect performance on the lexical decision task. To this end, participants were given five emotion measures and completed a lexical decision task with words varying on emotional experience.ResultsIt was found that participants who scored higher on the Positive and Negative Affect Schedule showed a weaker emotional experience effect. No other emotion measures interacted with emotional experience.ConclusionsThese results are predicted by and interpreted within a framework where the semantic representation of words includes emotion information that is grounded in experience of our internal states.

Sentiment Analysis of Social Network Comment Text Based on LSTM and Bert

This paper has the aim of solving problems in research studies on the analysis tasks of text emotion; the problems are the low utilization of text, the difficulty of effective information extraction, the failure of recognizing word polysemy with effectiveness. Thus, based on LSTM and Bert, the method of sentiment analysis on text is adopted. To be precise, word embedding of dataset in view of the skip-gram model is used for training course. In each sample, the word embeddings combine matric with the two-dimensional feature to be neural network input. Next, construction of analysis model for text sentiment combines Bert pre-training language model and long short-term memory (LSTM) network, using the word vector pre-trained by Bert instead of that trained in the traditional way to dynamically generate the semantic vector according to the word context. Finally, the semantic representation of words from text is improved by effectively identifying the polysemy of words, and the semantic vector is input into the LSTM to capture the semantic dependencies, thereby enhancing the ability to extract valid information. The Accuracy, Precision, Recall and F-Measure for the method of Bert–LSTM based on analysis of text sentiment are 0.89, 0.9, 0.84 and 0.87, indicating high value than the compared ones. Thus, the proposed method significantly outperforms the comparison methods in text sentiment analysis.

Using language models and ontology topology to perform semantic mapping of traits between biomedical datasets.

Human traits are typically represented in both the biomedical literature and large population studies as descriptive text strings. Whilst a number of ontologies exist, none of these perfectly represent the entire human phenome and exposome. Mapping trait names across large datasets is therefore time-consuming and challenging. Recent developments in language modelling have created new methods for semantic representation of words and phrases, and these methods offer new opportunities to map human trait names in the form of words and short phrases, both to ontologies and to each other. Here, we present a comparison between a range of established and more recent language modelling approaches for the task of mapping trait names from UK Biobank to the Experimental Factor Ontology (EFO), and also explore how they compare to each other in direct trait-to-trait mapping. In our analyses of 1191 traits from UK Biobank with manual EFO mappings, the BioSentVec model performed best at predicting these, matching 40.3% of the manual mappings correctly. The BlueBERT-EFO model (finetuned on EFO) performed nearly as well (38.8% of traits matching the manual mapping). In contrast, Levenshtein edit distance only mapped 22% of traits correctly. Pairwise mapping of traits to each other demonstrated that many of the models can accurately group similar traits based on their semantic similarity. Our code is available at https://github.com/MRCIEU/vectology.

Hyper-Graph Attention Based Federated Learning Method For Mental Health Detection.

Internet-Delivered Psychological Treatment (IDPT) has become necessary in the medical field. Deep neural networks (DNNs) require large, diverse patient populations to train models that achieve clinician-level performance. However, DNN models trained on limited datasets have poor clinical performance when used in a new location with different data. Thus, increasing the availability of diverse as well as distinct training data is vital. This study proposes a structural hypergraph as well as an emotional lexicon for word representation. An embedding model based on federated learning was developed for mental health symptom detection. The model treats text data as a collection of consecutive words. The model then learns a low-dimensional continuous vector while maintaining contextual linkage. The generated models with attention-based mechanisms as well as federated learning are then tested experimentally. Our strategy is suitable for vocabulary diversification, grammatical word representation, as well as dynamic lexicon analysis. The goal is to create semantic word representations using an attention network model. Later, clinical processes are used to mark the text by embedding it. Experimental results show the encoding of emotional words using the structural hypergraph. The 0.86 ROC was achieved using the bidirectional LSTM architecture with an attention mechanism.

Hyper-graph-based attention curriculum learning using a lexical algorithm for mental health

In this paper, we propose a structure hypergraph and an emotional lexicon for word representation. Our method can solve problems related to vocabulary size, grammatical representation of words, and the lack of an emotional lexicon. Natural Language Processing (NLP) and attention-based curriculum learning are then used in the developed model. The goal is to achieve semantic word representations using a graph model. Later, embedding is used to label the text using clinical procedures. The experimental results show the emotional word representation with the structure hypergraph. The bidirectional Long Short Term Memory (LSTM) architecture with an attention mechanism achieved a Receiver Operating Characteristic (ROC) value of 0.96. The learning method can help psychiatrists in note taking and contributes to the detection rate of depression symptoms.

A New Approach for Explicit Distributional Semantic Vector Representation Based on Dependency Distance and Bias Elimination

We consider the problem of the semantic representation of words based on explicit basis word vectors. We focus on explicit semantic vectors in which each basis vector is equivalent to a unique word. In this paper, we propose a new approach to represent semantic vectors explicitly. The approach consists of three steps: First, we introduce two new factors called word similarity (WS) and the number of zeroes (NZ) for choosing informative context words. Second, we propose a dependency distance method by incorporating the syntactic relations of the words in the sentences. This means that the method uses not only semantic information but also syntactic one. Third, we propose two shifting methods based on WF and WS factors to eliminate the bias of the PPMI matrix. In this work, we use the ukWaC corpus to select context words and compute co-occurrence matrices. Then, we examine the results on MEN, RG-65, SimLex-999, and WordSim353 test sets. We report the performance in comparison with a Baseline in which, the word frequency has been used. The Baseline method uses a window to fix the influence of the neighboring words. Our approach improves the Spearman correlation coefficient for MEN, RG-65, SimLex-999, and WordSim353 datasets by 3.87%, 10.25%, 5.04%, and 5.24%, respectively. So, the experiment results show that the proposed context words selection, incorporating dependency distance, and using the WF-based shifting method significantly improve the performance.

Content matters: Measures of contextual diversity must consider semantic content

Measures of contextual diversity seek to replace word frequency by counting the number of different contexts that a word occurs in rather than the total raw number of occurrences (Adelman, Brown, & Quesada, 2006). It has repeatedly been shown that contextual diversity measures outperform word frequency on word recognition datasets (Adelman & Brown, 2008; Brysbaert & New, 2009). Recently, Hollis (2020) demonstrated that the standard operationalization of contextual diversity as a document count accounts for relatively little unique variance over word frequency when other variables of contextual occurrences are controlled for. One aspect of the analysis conducted by Hollis (2020) that was not taken into account was the semantic content of the contexts that words occur in. Johns, Dye, and Jones (2020) and Johns (2021) have recently shown that defining linguistic contexts at larger, and more ecologically valid, levels lead to contextual diversity measures that provide very large improvements over word frequency, especially when implemented with principles from the Semantic Distinctiveness Model of Jones, Johns, and Recchia (2012). Across a series of simulations, we demonstrate that the advantages of contextual diversity measures are dependent upon the usage of semantic representations of words to determine the uniqueness of contextual occurrences, where unique contextual occurrences provide a greater impact to a word’s lexical strength than redundant contextual occurrences. The results of the simulations suggest that for better theoretical accounts of lexical strength to be developed, attention needs to be paid to the representation of linguistic contexts. Code and data associated with this article is available at https://osf.io/r5ec2/.

Schizophrenia Detection Using Machine Learning Approach from Social Media Content.

Schizophrenia is a severe mental disorder that ranks among the leading causes of disability worldwide. However, many cases of schizophrenia remain untreated due to failure to diagnose, self-denial, and social stigma. With the advent of social media, individuals suffering from schizophrenia share their mental health problems and seek support and treatment options. Machine learning approaches are increasingly used for detecting schizophrenia from social media posts. This study aims to determine whether machine learning could be effectively used to detect signs of schizophrenia in social media users by analyzing their social media texts. To this end, we collected posts from the social media platform Reddit focusing on schizophrenia, along with non-mental health related posts (fitness, jokes, meditation, parenting, relationships, and teaching) for the control group. We extracted linguistic features and content topics from the posts. Using supervised machine learning, we classified posts belonging to schizophrenia and interpreted important features to identify linguistic markers of schizophrenia. We applied unsupervised clustering to the features to uncover a coherent semantic representation of words in schizophrenia. We identified significant differences in linguistic features and topics including increased use of third person plural pronouns and negative emotion words and symptom-related topics. We distinguished schizophrenic from control posts with an accuracy of 96%. Finally, we found that coherent semantic groups of words were the key to detecting schizophrenia. Our findings suggest that machine learning approaches could help us understand the linguistic characteristics of schizophrenia and identify schizophrenia or otherwise at-risk individuals using social media texts.

Learning multimodal word representation with graph convolutional networks

Multimodal models have been proven to outperform text-based models on learning semantic word representations. According to psycholinguistic theory, there is a graphical relationship among the modalities of language, and in recent years, the graph convolution network (GCN) has been proven to have substantial advantages in the extraction of non-European spatial features. This inspires us to propose a new multimodal word representation model, namely, GCNW, which uses the graph convolutional network to incorporate the phonetic and syntactic information into the word representation. We use a greedy strategy to update the modality-relation matrix in the GCN, and we train the model through unsupervised learning. We evaluated the proposed model on multiple downstream NLP tasks, and various experimental results demonstrate that the GCNW outperforms strong unimodal baselines and state-of-the-art multimodal models. We make the source code of both models available to encourage reproducible research.

Hybrid Chinese text classification model based on pretraining model

In order to solve the problems of fuzzy boundary of Chinese words, polysemy of one word and the inability of traditional models to reflect the importance of each word in the text, a hybrid text classification model based on Ernie CNN and bilstm attention (mecba) is proposed. Firstly, Ernie model is used to generate the corresponding word vector, which can retain rich semantic information and enhance the semantic representation of words. Then, the word vector is input into CNN to extract local information, and the context feature is extracted by bilstm attention, and the output of the two is spliced; Finally, softmax classifier is used for classification. The experimental results show that the model is better than CNN, bilstm and other classification models, and can effectively improve the performance of Chinese text classification.

Probing Lexical Ambiguity: Word Vectors Encode Number and Relatedness of Senses.

Lexical ambiguity-the phenomenon of a single word having multiple, distinguishable senses-is pervasive in language. Both the degree of ambiguity of a word (roughly, its number of senses) and the relatedness of those senses have been found to have widespread effects on language acquisition and processing. Recently, distributional approaches to semantics, in which a word's meaning is determined by its contexts, have led to successful research quantifying the degree of ambiguity, but these measures have not distinguished between the ambiguity of words with multiple related senses versus multiple unrelated meanings. In this work, we present the first assessment of whether distributional meaning representations can capture the ambiguity structure of a word, including both the number and relatedness of senses. On a very large sample of English words, we find that some, but not all, distributional semantic representations that we test exhibit detectable differences between sets of monosemes (unambiguous words; N=964), polysemes (with multiple related senses; N=4,096), and homonyms (with multiple unrelated senses; N=355). Our findings begin to answer open questions from earlier work regarding whether distributional semantic representations of words, which successfully capture various semantic relationships, also reflect fine-grained aspects of meaning structure that influence human behavior. Our findings emphasize the importance of measuring whether proposed lexical representations capture such distinctions: In addition to standard benchmarks that test the similarity structure of distributional semantic models, we need to also consider whether they have cognitively plausible ambiguity structure.

The effects of word concreteness on cross-language neural pattern similarity during semantic categorization

Previous behavioral studies have suggested that word concreteness affects the similarity of semantic representations between native and second languages. However, it is unclear whether and how word concreteness influences the cross-language similarity in neural representations. In the present study, we quantified the neural pattern similarity between native and second languages by using representational similarity analysis, and examined the modulatory effect of word concreteness on cross-language similarity in neural representations. Twenty-six Chinese-English bilinguals performed a semantic decision task during fMRI scan. Results showed that concrete and abstract words elicited stronger activation in the right sensorimotor cortex for nonverbal processing and the left prefrontal cortex for verbal processing, respectively. Consistently, representational similarity analysis revealed that concrete and abstract words showed higher cross-language neural pattern similarity in those brain regions for nonverbal and verbal processing, respectively. More importantly, we found that the left middle temporal gyrus contained item-specific cross-language semantic representations for abstract words. These results suggest that word concreteness plays an important role in influencing cross-language similarity in neural representations. • RSA is used to quantify cross-language pattern similarity. • Concrete and abstract words elicit distinct neural activity patterns. • LMTG shows item-specific cross-language pattern similarity for abstract words.

Towards syntax-aware token embeddings

AbstractDistributional semantic word representations are at the basis of most modern NLP systems. Their usefulness has been proven across various tasks, particularly as inputs to deep learning models. Beyond that, much work investigated fine-tuning the generic word embeddings to leverage linguistic knowledge from large lexical resources. Some work investigated context-dependent word token embeddings motivated by word sense disambiguation, using sequential context and large lexical resources. More recently, acknowledging the need for an in-context representation of words, some work leveraged information derived from language modelling and large amounts of data to induce contextualised representations. In this paper, we investigate Syntax-Aware word Token Embeddings (SATokE) as a way to explicitly encode specific information derived from the linguistic analysis of a sentence in vectors which are input to a deep learning model. We propose an efficient unsupervised learning algorithm based on tensor factorisation for computing these token embeddings given an arbitrary graph of linguistic structure. Applying this method to syntactic dependency structures, we investigate the usefulness of such token representations as part of deep learning models of text understanding. We encode a sentence either by learning embeddings for its tokens and the relations between them from scratch or by leveraging pre-trained relation embeddings to infer token representations. Given sufficient data, the former is slightly more accurate than the latter, yet both provide more informative token embeddings than standard word representations, even when the word representations have been learned on the same type of context from larger corpora (namely pre-trained dependency-based word embeddings). We use a large set of supervised tasks and two major deep learning families of models for sentence understanding to evaluate our proposal. We empirically demonstrate the superiority of the token representations compared to popular distributional representations of words for various sentence and sentence pair classification tasks.

A semantic approach to extractive multi-document summarization: Applying sentence expansion for tuning of conceptual densities

Today, due to a vast amount of textual data, automated extractive text summarization is one of the most common and practical techniques for organizing information. Extractive summarization selects the most appropriate sentences from the text and provide a representative summary. The sentences, as individual textual units, usually are too short for major text processing techniques to provide appropriate performance. Hence, it seems vital to bridge the gap between short text units and conventional text processing methods.In this study, we propose a semantic method for implementing an extractive multi-document summarizer system by using a combination of statistical, machine learning based, and graph-based methods. It is a language-independent and unsupervised system. The proposed framework learns the semantic representation of words from a set of given documents via word2vec method. It expands each sentence through an innovative method with the most informative and the least redundant words related to the main topic of sentence. Sentence expansion implicitly performs word sense disambiguation and tunes the conceptual densities towards the central topic of each sentence. Then, it estimates the importance of sentences by using the graph representation of the documents. To identify the most important topics of the documents, we propose an inventive clustering approach. It autonomously determines the number of clusters and their initial centroids, and clusters sentences accordingly. The system selects the best sentences from appropriate clusters for the final summary with respect to information salience, minimum redundancy, and adequate coverage.A set of extensive experiments on DUC2002 and DUC2006 datasets was conducted for investigating the proposed scheme. Experimental results showed that the proposed sentence expansion algorithm and clustering approach could considerably enhance the performance of the summarization system. Also, comparative experiments demonstrated that the proposed framework outperforms most of the state-of-the-art summarizer systems and can impressively assist the task of extractive text summarization.

Predicting human odor perception represented by continuous values from mass spectra of essential oils resembling chemical mixtures.

There have been recent advances in predicting odor characteristics using molecular structure parameters of chemicals. Although the molecular structure parameters are available for each chemical, they cannot be used for chemical mixtures. This study will elucidate a computational method of predicting human odor perception from the mass spectra of chemical mixtures such as essential oils. Furthermore, a method for obtaining similarity among odor descriptors has been proposed although the dataset contains binary values only. When the database indicates a set of odor descriptors for one sample, only binary data are available and the correlation between the similar descriptors disappears. Thus, the prediction performance degrades for not considering the similarity among the odor descriptors. Since mass spectra dataset is highly dimensional, we use auto-encoder to learn the compressed representation from the mass spectra of essential oils in its bottleneck hidden layer and then accomplishes the hierarchical clustering to create odor descriptor groups with similar odor impressions using a matrix of continuous value-based correlation coefficient as well as natural language processing. This work will help to expatiate the process of overcoming binary value problem and find out the similarity among odor descriptors using machine learning with natural language semantic representation of words. To overcome the problem of disproportionate ratio of positive and negative class for both the continuous value-based correlation coefficient and word similarity based models, we use Synthetic Minority Oversampling Technique (SMOTE). This model allows us to predict human odor perception through computer simulations by forming odor descriptors group. Accordingly, this study demonstrates the feasibility of ensembling machine learning with natural language processing and SMOTE approach for predicting odor descriptor group from mass spectra of essential oils.

Automatic Annotation of Narrative Radiology Reports.

Narrative texts in electronic health records can be efficiently utilized for building decision support systems in the clinic, only if they are correctly interpreted automatically in accordance with a specified standard. This paper tackles the problem of developing an automated method of labeling free-form radiology reports, as a precursor for building query-capable report databases in hospitals. The analyzed dataset consists of 1295 radiology reports concerning the condition of a knee, retrospectively gathered at the Clinical Hospital Centre Rijeka, Croatia. Reports were manually labeled with one or more labels from a set of 10 most commonly occurring clinical conditions. After primary preprocessing of the texts, two sets of text classification methods were compared: (1) traditional classification models—Naive Bayes (NB), Logistic Regression (LR), Support Vector Machine (SVM), and Random Forests (RF)—coupled with Bag-of-Words (BoW) features (i.e., symbolic text representation) and (2) Convolutional Neural Network (CNN) coupled with dense word vectors (i.e., word embeddings as a semantic text representation) as input features. We resorted to nested 10-fold cross-validation to evaluate the performance of competing methods using accuracy, precision, recall, and score. The CNN with semantic word representations as input yielded the overall best performance, having a micro-averaged score of . The CNN classifier yielded particularly encouraging results for the most represented conditions: degenerative disease (), arthrosis (), and injury (). As a data-hungry deep learning model, the CNN, however, performed notably worse than the competing models on underrepresented classes with fewer training instances such as multicausal disease or metabolic disease. LR, RF, and SVM performed comparably well, with the obtained micro-averaged scores of , , and , respectively.

Learning Multimodal Word Representations by Explicitly Embedding Syntactic and Phonetic Information

Word embedding (i.e., word representation) transforms words into computable mathematical expressions (usually vectors) according to semantics. Compared with human semantic representation, these purely text-based models are severely deficient because they lack perceptual information attached to the physical world. This observation promotes the development of multimodal word representation models. Multimodal models have been proven to outperform text-based models on learning semantic word representations, and almost all previous multimodal models only focus on introducing perceptual information. However, it is obvious that syntactic information can effectively improve the performance of multimodal models on downstream tasks. Therefore, this article proposes an effective multimodal word representation model that uses two gate mechanisms to explicitly embed syntactic and phonetic information into multimodal representations and uses supervised learning to train the model. We select Chinese and English as examples and evaluate the model using several downstream tasks. The results show that our approach outperforms the existing models. We have made the source code of the model available to encourage reproducible research.

Incorporating Semantic Word Representations into Query Expansion for Microblog Information Retrieval

Microblog information retrieval has attracted much attention of researchers to capture the desired information in daily communications on social networks. Since the contents of microblogs are always non-standardized and flexible, including many popular Internet expressions, the retrieval accuracy of microblogs has much room for improvement. To enhance microblog information retrieval, we propose a novel query expansion method to enrich user queries with semantic word representations. In our method, we use a neural network model to map each word in the corpus to a low-dimensional vector representation. The mapped word vectors satisfy the algebraic vector addition operation, and the new vector obtained by the addition operation can express some common attributes of the two words. In this sense, we represent keywords in user queries as vectors, sum all the keyword vectors, and use the obtained query vectors to select the expansion words. In addition, we also combine the traditional pseudo-relevance feedback query expansion method with the proposed query expansion method. Experimental results show that the proposed method is effective and reduces noises in the expanded query, which improves the accuracy of microblog retrieval.