Word Embedding Technique Research Articles

Identifying Phage Sequences From Metagenomic Data Using Deep Neural Network With Word Embedding and Attention Mechanism.

Phages are the functional viruses that infect bacteria and they play important roles in microbial communities and ecosystems. Phage research has attracted great attention due to the wide applications of phage therapy in treating bacterial infection in recent years. Metagenomics sequencing technique can sequence microbial communities directly from an environmental sample. Identifying phage sequences from metagenomic data is a vital step in the downstream of phage analysis. However, the existing methods for phage identification suffer from some limitations in the utilization of the phage feature for prediction, and therefore their prediction performance still need to be improved further. In this article, we propose a novel deep neural network (called MetaPhaPred) for identifying phages from metagenomic data. In MetaPhaPred, we first use a word embedding technique to encode the metagenomic sequences into word vectors, extracting the latent feature vectors of DNA words. Then, we design a deep neural network with a convolutional neural network (CNN) to capture the feature maps in sequences, and with a bi-directional long short-term memory network (Bi-LSTM) to capture the long-term dependencies between features from both forward and backward directions. The feature map consists of a set of feature patterns, each of which is the weighted feature extracted by a convolution filter with convolution kernels in the CNN slide along the input feature vectors. Next, an attention mechanism is used to enhance contributions of important features. Experimental results on both simulated and real metagenomic data with different lengths demonstrate the superiority of the proposed MetaPhaPred over the state-of-the-art methods in identifying phage sequences.

Arabic Toxic Tweet Classification: Leveraging the AraBERT Model

Social media platforms have become the primary means of communication and information sharing, facilitating interactive exchanges among users. Unfortunately, these platforms also witness the dissemination of inappropriate and toxic content, including hate speech and insults. While significant efforts have been made to classify toxic content in the English language, the same level of attention has not been given to Arabic texts. This study addresses this gap by constructing a standardized Arabic dataset specifically designed for toxic tweet classification. The dataset is annotated automatically using Google’s Perspective API and the expertise of three native Arabic speakers and linguists. To evaluate the performance of different models, we conduct a series of experiments using seven models: long short-term memory (LSTM), bidirectional LSTM, a convolutional neural network, a gated recurrent unit (GRU), bidirectional GRU, multilingual bidirectional encoder representations from transformers, and AraBERT. Additionally, we employ word embedding techniques. Our experimental findings demonstrate that the fine-tuned AraBERT model surpasses the performance of other models, achieving an impressive accuracy of 0.9960. Notably, this accuracy value outperforms similar approaches reported in recent literature. This study represents a significant advancement in Arabic toxic tweet classification, shedding light on the importance of addressing toxicity in social media platforms while considering diverse languages and cultures.

A detailed review on word embedding techniques with emphasis on word2vec

A detailed review on word embedding techniques with emphasis on word2vec

Automatic Construction of Educational Knowledge Graphs: A Word Embedding-Based Approach

Knowledge graphs (KGs) are widely used in the education domain to offer learners a semantic representation of domain concepts from educational content and their relations, termed as educational knowledge graphs (EduKGs). Previous studies on EduKGs have incorporated concept extraction and weighting modules. However, these studies face limitations in terms of accuracy and performance. To address these challenges, this work aims to improve the concept extraction and weighting mechanisms by leveraging state-of-the-art word and sentence embedding techniques. Concretely, we enhance the SIFRank keyphrase extraction method by using SqueezeBERT and we propose a concept-weighting strategy based on SBERT. Furthermore, we conduct extensive experiments on different datasets, demonstrating significant improvements over several state-of-the-art keyphrase extraction and concept-weighting techniques.

Extractive Summarization of Telugu Text Using Modified Text Rank and Maximum Marginal Relevance

With the rapid growth of digital content, there is a need for an automatic text summarizer to provide short text from a long text document. Many research works have been presented for extractive text summarization (ETS). This article mainly focuses on the graph-based ETS approach for multiple Telugu text documents. A modified Text-Rank algorithm is employed with the noun and verb count of each sentence in the text as the initial score of each node. To get the optimal features, a novel feature selection algorithm called improved Flamingo Search Algorithm is proposed in this article. Though graph-based ETS is an important approach, the generated summaries are redundant. To reduce the redundancy in the generated summary, maximum marginal relevance is combined with the modified Text-Rank. Different word-embedding techniques such as Fast-Text, Word2vec, TF-IDF, and one-hot encoding are utilized to experiment with the proposed approach. The performance of the proposed text summarization approach is evaluated with BLEU and ROUGE in terms of F-measure, precision, and recall.

Statement Recognition of Access Control Policies in IoT Networks.

Access Control Policies (ACPs) are essential for ensuring secure and authorized access to resources in IoT networks. Recognizing these policies involves identifying relevant statements within project documents expressed in natural language. While current research focuses on improving recognition accuracy through algorithm enhancements, the challenge of limited labeled data from individual clients is often overlooked, which impedes the training of highly accurate models. To address this issue and harness the potential of IoT networks, this paper presents FL-Bert-BiLSTM, a novel model that combines federated learning and pre-trained word embedding techniques for access control policy recognition. By leveraging the capabilities of IoT networks, the proposed model enables real-time and distributed training on IoT devices, effectively mitigating the scarcity of labeled data and enhancing accessibility for IoT applications. Additionally, the model incorporates pre-trained word embeddings to leverage the semantic information embedded in textual data, resulting in improved accuracy for access control policy recognition. Experimental results substantiate that the proposed model not only enhances accuracy and generalization capability but also preserves data privacy, making it well-suited for secure and efficient access control in IoT networks.

Distributed Representations of Wi-Fi Fingerprints from Non-Contextual Text-Embedding Techniques with Applications in Crowdsourcing Zone-Level Localization

Over the past decade, indoor localization systems have gained increasing attention and found widespread applications in commercial and research environments. Specifically, a Wi-Fi fingerprint-based system offers a low-cost solution over its counterparts such as Bluetooth, ultra-wideband (UWB), and radio frequency identification (RFID) technologies due to the ubiquity of Wi-Fi access points (WAPs) in most buildings. However, the main disadvantage of the fingerprint-based system is intensive survey effort required during system initialization and maintenance. This work explores a solution to alleviate this limitation by considering a crowdsourcing approach for zone-level localization. Instead of relying only on the labelled fingerprint data from trained surveyors, this approach uses the more-attainable unlabelled fingerprint data collected by participating volunteers. This unlabelled data is then used to augment the survey data in a process called pseudo labelling, forming a more comprehensive training dataset for subsequent localization tasks; this semi-supervised approach allows for minimal survey effort during system initialization and maintenance. To enable such solution, this work introduces a novel approach of employing non-contextual word embedding techniques to construct distributed vector representations of fingerprint data to overcome 3 challenges; (a) high memory requirement in the downstream tasks due to high-dimensional non-distributed vector representations from the “standard” vector transformation, (b) inclusion of an arbitrary value that represents missing WAPs which can affect the performance of the downstream localization tasks in a non-transparent manner, and most importantly, (c) poor pseudo-labelling and semi-supervised zone-prediction performances due to poor data separability in a feature space. The choice of the non-contextual text-embedding techniques, as opposed to the contextual counterparts, leads to less computational requirement in model training and distributed-representation generation due to simpler model architectures (no deep learning) and no requirement for pre-trained model during distributed-representation generation. To this end, we considered non-contextual word embedding techniques commonly used in natural language processing such as Word2Vec, GloVe, and Doc2Vec in the distributed-representation transformation, and compared the resulting downstream performances with those from well-recognized dimensionality reduction techniques such as PCA, Isomap, and UMAP. The results show that Word2Vec and GloVe transformations outperform other types of transformations in terms of separability in fingerprint representations, pseudo-labelling performance, and semi-supervised zone-prediction accuracy. Together with the promising robustness property against potential data inhomogeneity, Word2Vec and GloVe transformations are the recommended transformation processes for constructing vector representations of fingerprints in crowdsourcing zone-level localization. HIGHLIGHTS This work introduces a novel approach of employing non-contexual word-embedding techniques to construct distributed vector representations of Wi-Fi fingerprint data to facilitate pseudo-labelling and semi-supervised zone-prediction tasks in crowdsourcing zone-level localization The benefits of employing word-embedding techniques are (a) lower memory requirement in the downstream tasks due to distributed vector representations (b) no inclusion of an arbitrary value that represents missing WAPs which can affect the performance of the downstream localization tasks in a non-transparent manner (c) improved pseudo-labelling and semi-supervised zone-prediction performances due to improved data separability in a feature space The benefit of employing non-contextual techniques, as opposed to the contextual counterparts, is less computational requirement in model training and distributed-representation generation due to simpler model architectures (no deep learning) and no requirement for pre-trained model during distributed-representation generation The results show that Word2Vec and GloVe transformations outperform other types of transformations in terms of separability in fingerprint representations, pseudo-labelling performance, and semi-supervised zone-prediction accuracy GRAPHICAL ABSTRACT

Low-Resource Language Information Processing using Dwarf Mongoose Optimization with Deep Learning Based Sentiment Classification

Asian and low-resource language information processing refers to the field of computational linguistics that aims to develop natural language processing (NLP) technologies for languages that have fewer available language resources or are less commonly spoken. This is an important field of study because many languages in Asia and other parts of the world are underrepresented in the field of NLP, which may limit access to information and technology for speakers of these languages. The growing volume of user-generated content on the web has made sentiment analysis (SA) a significant tool for extracting data regarding human emotional states. Twitter sentiment detectors provide a superior solution for assessing the quality of products and services compared to other conventional technologies. The detection performance and classifier accuracy of SA, which can be highly dependent on classifier methods and the quality of input features have been utilised. Deep learning (DL) methods use distinct techniques to extract data from raw data such as tweets or texts and represent them in different forms of models. Therefore, this article presents a Dwarf Mongoose Optimization with Deep Learning-Based Twitter Sentiment Classification (DMODL-TSC) technique to classify sentiments based on tweets. The presented DMODL-TSC technique leverages the concepts of natural language processing (NLP) and DL. Primarily, the raw tweets are preprocessed to transform them into a useful format. Next, the DMODL-TSC technique uses the advanced FastText word embedding technique. Moreover, the bidirectional recurrent neural network (BiRNN) method is utilized for the recognition of sentiments. Finally, the DMO technique is utilized for the optimal hyperparameter optimization of the BiRNN method, which leads to effective classification performance. The comprehensive result examination of the DMODL-TSC system was tested on three datasets, and the obtained outcomes illustrate the supremacy of the DMODL-TSC approach.

Describe the house and I will tell you the price: House price prediction with textual description data

Abstract House price prediction is an important problem that could benefit home buyers and sellers. Traditional models for house price prediction use numerical attributes such as the number of rooms but disregard the house description text. The recent developments in text processing suggest these can be valuable attributes, which motivated us to use house descriptions. This paper focuses on the house asking/advertising price and studies the impact of using house description texts to predict the final house price. To achieve this, we collected a large and diverse set of attributes on house postings, including the house advertising price. Then, we compare the performance of three scenarios: using only the house description, only numeric attributes, or both. We processed the description text through three word embedding techniques: TF-IDF, Word2Vec, and BERT. Four regression algorithms are trained using only textual data, non-textual data, or both. Our results show that by using exclusively the description data with Word2Vec and a Deep Learning model, we can achieve good performance. However, the best overall performance is obtained when using both textual and non-textual features. An $R^2$ of 0.7904 is achieved by the deep learning model using only description data on the testing data. This clearly indicates that using the house description text alone is a strong predictor for the house price. However, when observing the RMSE on the test data, the best model was gradient boosting using both numeric and description data. Overall, we observe that combining the textual and non-textual features improves the learned model and provides performance benefits when compared against using only one of the feature types. We also provide a freely available application for house price prediction, which is solely based on a house text description and uses our final developed model with Word2Vec and Deep Learning to predict the house price.

CoBiCo: A model using multi-stage ConvNet with attention-based Bi-LSTM for efficient sentiment classification

The rapid growth of social media and specialized websites that provide critical product reviews has resulted in a massive collection of information for customers worldwide. These data could contain a wealth of information, such as product reviews, market forecasting, and the polarity of sentiments. In these challenges, machine learning and deep learning algorithms give the necessary capabilities for sentiment analysis. In today’s competitive markets, it’s critical to grasp reviewer opinions and sentiments by extracting and analyzing their characteristics. The research aims to develop an optimised model for evaluating sentiments and categorising them into proper categories. This research proposes a unique, novel hybridised model that integrates the advantages of deep learning methods Dual LSTM (Long Short Term Memory) and CNN (Convolution Neural Network) with word embedding technique. The performance of different word embedding techniques is compared to select the best embedding for the implementation in the proposed model. Furthermore, a multi-convolution approach with attention-oriented BiLSTM is applied. To test the validity of the performance of the proposed model, standard metrics were applied. The outcome indicates that the suggested model achieves a significantly improved accuracy of 96.56%, superior to other models.

Robust Malware identification via deep temporal convolutional network with symmetric cross entropy learning

AbstractRecent developments in the field of Internet of things (IoT) have aroused growing attention to the security of smart devices. Specifically, there is an increasing number of malicious software (Malware) on IoT systems. Nowadays, researchers have made many efforts concerning supervised machine learning methods to identify malicious attacks. High‐quality labels are of great importance for supervised machine learning, but noises widely exist due to the non‐deterministic production environment. Therefore, learning from noisy labels is significant for machine learning‐enabled Malware identification. In this study, motivated by the symmetric cross entropy with satisfactory noise robustness, the authors propose a robust Malware identification method using temporal convolutional network (TCN). Moreover, word embedding techniques are generally utilised to understand the contextual relationship between the input operation code (opcode) and application programming interface function names. Here, considering the numerous unlabelled samples in real‐world intelligent environments, the authors pre‐train the TCN model on an unlabelled set using a word embedding method, that is, Word2Vec. In the experiments, the proposed method is compared with several traditional statistical methods and more recent neural networks on a synthetic Malware dataset and a real‐world dataset. The performance comparisons demonstrate the better performance and noise robustness of their proposed method, especially that the proposed method can yield the best identification accuracy of 98.75% in real‐world scenarios.

Detection of Software Security Weaknesses Using Cross-Language Source Code Representation (CLaSCoRe)

The research presented in the paper aims at increasing the capacity to identify security weaknesses in programming languages that are less supported by specialized security analysis tools, based on the knowledge gathered from securing the popular ones, for which security experts, scanners, and labeled datasets are, in general, available. This goal is vital in reducing the overall exposure of software applications. We propose a solution to expand the capabilities of security gaps detection to downstream languages, influenced by their more popular “ancestors” from the programming languages’ evolutionary tree, using language keyword tokenization and clustering based on word embedding techniques. We show that after training a machine learning algorithm on C, C++, and Java applications developed by a community of programmers with similar behavior of writing code, we can detect, with acceptable accuracy, similar vulnerabilities in C# source code written by the same community. To achieve this, we propose a core cross-language representation of source code, optimized for security weaknesses classifiers, named CLaSCoRe. Using this method, we can achieve zero-shot vulnerability detection—in our case, without using any training data with C# source code.

A comparison of word embedding-based extraction feature techniques and deep learning models of natural disaster messages classification

The research aims to compare the classification performance of natural disaster messages classification from Twitter. The research experiment covers the analysis of three-word embedding-based extraction feature techniques and five different models of deep learning. The word embedding techniques that are used in this experiment are Word2Vec, fastText, and Glove. The experiment uses five deep learning models, namely three models of different dimensions of Convolutional Neural Network (1D CNN, 2D CNN, 3D CNN), Long Short-Term Memory Network (LSTM), and Bidirectional Encoder Representations for Transformer (BERT). The models are tested on four natural disaster messages datasets: earthquakes, floods, forest fires, and hurricanes. Those models are tested for classification performance

A comparative study on word embedding techniques for suicide prediction on COVID-19 tweets using deep learning models

A comparative study on word embedding techniques for suicide prediction on COVID-19 tweets using deep learning models

A Low-Complexity Deep Learning Model for Predicting Targeted Sequencing Depth from Probe Sequence

Targeted sequencing has been widely utilized for genomic molecular diagnostics and the emerging DNA data storage paradigm. However, the probe sequences used to enrich regions of interest have different hybridization kinetic properties, resulting in poor sequencing uniformity and setting limitations for the large-scale application of the technology. Here, a low-complexity deep learning model is proposed for prediction of sequencing depth from probe sequences. To capture the representation of probe and target sequences, we utilized a sequence-encoding model that incorporates k-mer and word embedding techniques, providing a streamlined alternative to the intricate computations involved in biochemical feature analysis. We employed bidirectional long short-term memory (Bi-LSTM) to effectively capture both long-range and short-range interactions within the representation. Furthermore, the attention mechanism was adopted to identify pivotal regions in the sequences that significantly influence sequencing depth. The ratio of the predicted sequencing depth to the actual sequencing depth was in the interval of 1/3—3 as the evaluation metric of model accuracy. The prediction accuracy was 94.3% in the human single-nucleotide polymorphism (SNP) panel and 99.7% in the synthetic DNA information storage sequence (SynDNA) panel. Our model substantially reduced data processing time (from 334 min to 4 min of CPU time in the SNP panel) and model parameters (from 300 k to 70 k) compared with the baseline model.

Comparison of Data Models for Unsupervised Twitter Sentiment Analysis

"Identifying the sentiment of collected tweets has become a challenging and interesting task. In addition, mining and defining relevant features that can improve the quality of a classification system is crucial. The data modeling phase is fundamental for the whole process since it can reveal hidden information from the textual inputs. Two models are defined in the presented paper, considering Twitter-specific concepts: a hashtagbased representation and a text-based one. These models will be compared and integrated into an unsupervised system that determines groups of tweets based on sentiment labels (positive and negative). Moreover, wordembedding techniques (TF-IDF and frequency vectors) are used to convert the representations into a numeric input needed for the clustering methods. The experimental results show good values for Silhouette and Davies-Bouldin measures in the unsupervised environment. A detailed investigation is presented considering several items (dataset, clustering method, data representation, or word embeddings) for checking the best setup for increasing the quality of detecting the sentiment from Twitter’s messages. The analysis and conclusions show that the first results can be considered for more complex experiments. Keywords: Sentiment Analysis, Twitter, Data Representation, Hashtags, Clustering. "

Fake news detection based on a hybrid BERT and LightGBM models

With the rapid growth of social networks and technology, knowing what news to believe and what not to believe become a challenge in this digital era. Fake news is defined as provably erroneous information transmitted intending to defraud. This kind of misinformation poses a serious threat to social cohesion and well-being, since it fosters political polarisation and can destabilize trust in the government or the service provided. As a result, fake news detection has emerged as an important field of study, with the goal of identifying whether a certain piece of content is real or fake. In this paper, we propose a novel hybrid fake news detection system that combines a BERT-based (bidirectional encoder representations from transformers) with a light gradient boosting machine (LightGBM) model. We compare the performance of the proposed method to four different classification approaches using different word embedding techniques on three real-world fake news datasets to validate the performance of the proposed method compared to other methods. The proposed method is evaluated to detect fake news based on the headline-only or full text of the news content. The results show the superiority of the proposed method for fake news detection compared to many state-of-the-art methods.

DiMo: discovery of microRNA motifs using deep learning and motif embedding.

MicroRNAs are small regulatory RNAs that decrease gene expression after transcription in various biological disciplines. In bioinformatics, identifying microRNAs and predicting their functionalities is critical. Finding motifs is one of the most well-known and important methods for identifying the functionalities of microRNAs. Several motif discovery techniques have been proposed, some of which rely on artificial intelligence-based techniques. However, in the case of few or no training data, their accuracy is low. In this research, we propose a new computational approach, called DiMo, for identifying motifs in microRNAs and generally macromolecules of small length. We employ word embedding techniques and deep learning models to improve the accuracy of motif discovery results. Also, we rely on transfer learning models to pre-train a model and use it in cases of a lack of (enough) training data. We compare our approach with five state-of-the-art works using three real-world datasets. DiMo outperforms the selected related works in terms of precision, recall, accuracy and f1-score.

An analysis of the thematic evolution of ageing and healthcare expenditure using word embedding: A scoping review of policy implications

Thanks to the advances in public health and medical sciences, the increasing of elderly has become a structural phenomenon in Europe and the rest of the World. This demographic age shift highlighted severe doubts about the sustainability of healthcare systems which see adult people as a burden rather than, if they are healthy and active, as an advantage for a country’s economic development. Safeguarding older people from infections is crucial, but supporting them during health emergencies is even more critical. Older adults, for example, became more at risk of death during the COVID-19 pandemic. Thus, the balance previously achieved between age-related diseases and healthy quality of life went under higher pressure. As a result, the potential impact of ageing on healthcare expenditure is increasingly debated. Starting from this background, we propose a bibliometric study based on word embedding techniques to identify the drivers that could influence this relationship and track their evolution in the health policy debate. The main findings of our research underline that given the upward impact on the growth in health spending due to population ageing, policies oriented to improve their access to health services through new technologies could better meet the elderly needs, moderating the cost growth. Protecting the health of older adults represents, in fact, a challenge for governments as well as proof of the quality of health systems management, especially during severe outbreaks such as the one caused by the coronavirus.

Framing climate change in Nature and Science editorials: applications of supervised and unsupervised text categorization

Hulme et al. (Nat Clim Change, 8:515–521, 2018) manually coded ‘frames’ in 490 Nature and Science editorials (1966–2016) they found relevant for climate change. We produced a digital version of the corpus and conducted a set of experiments: We explored many variants of supervised categorization for automatically reproducing the manual frame coding, and we ran an interactive variant of topic modeling. In both approaches, we made use of word embedding techniques for representing text documents. Supervised classification yielded F1-scores of up to 0.91 (for the best category) and 0.68 overall, and it led to insights regarding the relation between ‘topic’ and ‘framing’. The topic modeling algorithm was able to reproduce central trends in the temporal analysis of framing that was presented by Hulme et al. based on their manual work.