Embedding Technique Research Articles

Novel approach for Arabic fake news classification using embedding from large language features with CNN-LSTM ensemble model and explainable AI

The widespread fake news challenges the management of low-quality information, making effective detection strategies necessary. This study addresses this critical issue by advancing fake news detection in Arabic and overcoming limitations in existing approaches. Deep learning models, Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM), EfficientNetB4, Inception, Xception, ResNet, ConvLSTM and a novel voting ensemble framework combining CNN and LSTM are employed for text classification. The proposed framework integrates the ELMO word embedding technique having contextual representation capabilities, which is compared with GloVe, BERT, FastText and FastText subwords. Comprehensive experiments demonstrate that the proposed voting ensemble, combined with ELMo word embeddings, consistently outperforms previous approaches. It achieves an accuracy of 98.42%, precision of 98.54%, recall of 99.5%, and an F1 score of 98.93%, offering an efficient and highly effective solution for text classification tasks.The proposed framework benchmark against state-of-the-art transformer architectures, including BERT and RoBERTa, demonstrates competitive performance with significantly reduced inference time and enhanced interpretability accompanied by a 5-fold cross-validation technique. Furthermore, this research utilizes the LIME XAI technique to provide deeper insights into the contribution of each feature in predicting a specific target class. These findings show the proposed framework’s effectiveness in dealing with the issues of detecting false news, particularly in Arabic text. By generating higher performance metrics and displaying comparable results, this work opens the way for more reliable and interpretable text classification solutions.

Identifying the factors influencing the development of bilateral investment treaties with health safeguards: a Machine Learning-based link prediction approach

A network analysis approach, complemented by machine learning (ML) techniques, is applied to analyse the factors influencing Bilateral Investment Treaties (BITs) at the country level. Using the Electronic Database of Investment Treaties, BITs with health safeguards from 167 countries were charted, resulting in 534 connections with countries as nodes and their BITs as edges. Network analysis found that, on average, a country established BITs with six other nations. Additionally, we used node embedding techniques to generate features from the network, such as the Jaccard coefficient, resource allocation, and Adamic Adar for downstream link prediction. This study employed five tree-based ML models to predict future BIT formations with health inclusion. The eXtreme Gradient Boosting model proved to be superior, achieving a 64.02% accuracy rate. Notably, the Common Neighbor centrality feature and the Capital Account Balance Ratio emerged as influential factors in creating new BITs with health inclusions. Beyond economic considerations, our study highlighted a vital intersection: the nexus between BITs, economic growth, and public health policies. In essence, this research underscores the importance of safeguarding public health in BITs and showcases the potential of ML in understanding the intricacies of international treaties.

Phase transitions of correlated systems from graph neural networks with quantum embedding techniques

Phase transitions of correlated systems from graph neural networks with quantum embedding techniques

Semantic Based Image Indexing using Deep Learning

This paper introduces a semantic-based image indexing system utilizing a custom Convolutional Neural Network (CNN) for feature extraction and semantic embedding techniques for understanding image content. Traditional image indexing methods rely heavily on low-level visual features, often resulting in inaccurate or irrelevant results. By leveraging deep learning, our proposed system bridges this gap, allowing high-level semantic features to guide indexing and retrieval. Tested on the CIFAR-10 dataset, our approach demonstrates a significant improvement in precision, recall, and overall retrieval performance, showcasing its potential in real-world applications

Customer Churn Prediction Approach Based on LLM Embeddings and Logistic Regression

Nowadays, predicting customer churn is essential for the success of any company. Loyal customers generate continuous revenue streams, resulting in long-term success and growth. Moreover, companies are increasingly prioritizing the retention of existing customers due to the higher costs associated with attracting new ones. Consequently, there has been a growing demand for advanced methods aimed at enhancing customer loyalty and satisfaction, as well as predicting churners. In our work, we focused on building a robust churn prediction model for the telecommunications industry based on large embeddings from large language models and logistic regression to accurately identify churners. We conducted extensive experiments using a range of embedding techniques, including OpenAI Text-embedding, Google Gemini Text Embedding, bidirectional encoder representations from transformers (BERT), Sentence-Transformers, Sent2vec, and Doc2vec, to extract meaningful features. Additionally, we tested various classifiers, including logistic regression, support vector machine, random forest, K-nearest neighbors, multilayer perceptron, naive Bayes, decision tree, and zero-shot classification, to build a robust model capable of making accurate predictions. The best-performing model in our experiments is the logistic regression classifier, which we trained using the extracted feature from the OpenAI Text-embedding-ada-002 model, achieving an accuracy of 89%. The proposed model demonstrates a high discriminative ability between churning and loyal customers.

Approximating the semantic space: word embedding techniques in psychiatric speech analysis

Large language models provide high-dimensional representations (embeddings) of word meaning, which allow quantifying changes in the geometry of the semantic space in mental disorders. A pattern of a more condensed (‘shrinking’) semantic space marked by an increase in mean semantic similarity between words has been recently documented in psychosis across several languages. We aimed to explore this pattern further in picture descriptions provided by a transdiagnostic German sample of patients with schizophrenia spectrum disorders (SSD) (n = 42), major depression (MDD, n = 43), and healthy controls (n = 44). Compared to controls, both clinical groups showed more restricted dynamic navigational patterns as captured by the time series of semantic distances crossed, while also showing differential patterns in the total distances and trajectories navigated. These findings demonstrate alterations centred on the dynamics of the flow of meaning across the semantic space in SSD and MDD, preserving previous indications towards a shrinking semantic space in both cases.

Bangla hate speech detection by embedding and hybrid machine learning algorithms

The worrisome increase in hate speech in Bangladesh, driven by the rapidly expanding social media user base, has adversely affected cybersecurity and online safety. Most hate speech detection technologies overlook less commonly spoken languages, such as Bangla, in favour of more widely used ones. This project aims to address the gap by identifying hate speech in Bangla through the utilisation of sophisticated word embedding techniques (Word2Vec, GloVe, and FastText) and machine learning algorithms (Multinomial Naive Bayes, Random Forest, K-Nearest Neighbours, and Extreme Gradient Boosting). We used a dataset of 30,000 annotated messages, encompassing both positive and hate speech, to train and test the models. FastText combined with hybrid RF and SVM yielded the highest performance among the models, with an accuracy of 96.03%. The system's generalisability and usefulness were evident when it identified hate speech not included in the training data. Enhancing the identification of harmful content in Bangla is essential for the burgeoning digital ecosystem in Bangladesh, hence contributing to cybersecurity. This endeavour establishes a foundation for future study by efficiently employing advanced word embedding and machine learning techniques to identify hate speech in Bangla. It could significantly influence the advancement of more secure digital environments and the overall condition of internet security.

Highly Sensitive and Stable In Situ Acetylene Detection in Transformer Oil Using Polyimide-Embedded Carbon Nanotubes.

This study presents an acetylene gas sensor capable of in situ monitoring transformer oils. This sensor utilizes carbon nanotubes (CNTs) embedded in polyimide (PI) synthesized by floating catalyst chemical vapor deposition. Unlike conventional sensors that target hydrocarbon gases dissolved in oil and measure the gas extracted from the oil, the proposed CNT-PI sensor detects gas within the oil in real time. The PI embedding technique effectively anchors and shields the CNT network against fluidic damage, ensuring stable sensing performance over 6 months, even under friction stress caused by oil convection. Decorating CNTs with gold nanoparticles further enhances the sensitivity and response of the sensor. The sensor achieves a high response (10.5% at 30ppm) and fast response/recovery times (28 s/77 s), Furthermore, the sensor demonstrates good response (10.4% at 30ppm) and moderate response/recovery times (444 s/670 s) in an oil medium, which qualifies for industrial applications. Additionally, a CNT-PI-based heater is integrated into the sensor as a multilayer component, maintaining an optimal operating temperature of 90 °C. The CNT-PI sensor demonstrates consistent gas-sensing performance even after 10,000 bending cycles and exhibits superior characteristics, indicating its compatibility with various forms of transformers.

A novel deep neural network-based technique for network embedding.

In this paper, the graph segmentation (GSeg) method has been proposed. This solution is a novel graph neural network framework for network embedding that leverages the inherent characteristics of nodes and the underlying local network topology. The key innovation of GSeg lies in its encoder-decoder architecture, which is specifically designed to preserve the network's structural properties. The key contributions of GSeg are: (1) a novel graph neural network architecture that effectively captures local and global network structures, and (2) a robust node representation learning approach that achieves superior performance in various network analysis tasks. The methodology employed in our study involves the utilization of a graph neural network framework for the acquisition of node representations. The design leverages the inherent characteristics of nodes and the underlying local network topology. To enhance the architectural framework of encoder- decoder networks, the GSeg model is specifically devised to exhibit a structural resemblance to the SegNet model. The obtained empirical results on multiple benchmark datasets demonstrate that the GSeg outperforms existing state-of-the-art methods in terms of network structure preservation and prediction accuracy for downstream tasks. The proposed technique has potential utility across a range of practical applications in the real world.

An Empirical Exploration of the Effectiveness of Word Embedding Techniques for Emotion Recognition

An Empirical Exploration of the Effectiveness of Word Embedding Techniques for Emotion Recognition

Developing Semantic Textual Similarity for Guragigna Language Using Deep Learning Approach

Semantic Similarity is one of the highest levels of NLP. STS has significant advantages in NLP applications like information retrieval, information extraction, text summarization, data mining, machine translation, and other tasks. This research aims to present a deep learning approach for capturing semantic textual similarity (STS) in the Guragigna. The methodology involves collecting a Guragigna language corpus and preprocessing the text data and text representation is done using the Universal Sentence Encoder (USE), along with word embedding techniques including Word2Vec and GloVe and mean Square Error (MSE) is used to measure the performance. In the experimentation phase, models like LSTM, Bidirectional RNN, GRU, and Stacked RNN are trained and evaluated using different embedding techniques. The results demonstrate the efficacy of the developed models in capturing semantic textual similarity in the Guragigna language. Across different embedding techniques, including Word2Vec, GloVe, and USE, the Bidirectional RNN model with USE embedding achieves the lowest MSE of 0.0950 and the highest accuracy of 0.9244. GloVe and Word2Vec embedding also show competitive performance with slightly higher MSE and lower accuracy. The Universal Sentence Encoder consistently emerges as the top-performing embedding across all RNN architectures. The research results demonstrate the effectiveness of LSTM, GRU, Bi RNN, and Stacked RNN models in measuring semantic textual similarity in the Guragigna language.

Credibility-based knowledge graph embedding for identifying social brand advocates.

Brand advocates, characterized by their enthusiasm for promoting a brand without incentives, play a crucial role in driving positive word-of-mouth (WOM) and influencing potential customers. However, there is a notable lack of intelligent systems capable of accurately identifying online advocates based on their social interactions with brands. Knowledge Graphs (KGs) offer structured and factual representations of human knowledge, providing a potential solution to gain holistic insights into customer preferences and interactions with a brand. This study presents a novel framework that leverages KG construction and embedding techniques to identify brand advocates accurately. By harnessing the power of KGs, our framework enhances the accuracy and efficiency of identifying and understanding brand advocates, providing valuable insights into customer advocacy dynamics in the online realm. Moreover, we address the critical aspect of social credibility, which significantly influences the impact of advocacy efforts. Incorporating social credibility analysis into our framework allows businesses to identify and mitigate spammers, preserving authenticity and customer trust. To achieve this, we incorporate and extend DSpamOnto, a specialized ontology designed to identify social spam, with a focus on the social commerce domain. Additionally, we employ cutting-edge embedding techniques to map the KG into a low-dimensional vector space, enabling effective link prediction, clustering, and visualization. Through a rigorous evaluation process, we demonstrate the effectiveness and performance of our proposed framework, highlighting its potential to empower businesses in cultivating brand advocates and driving meaningful customer engagement strategies.

Text-Mining-Based Non-Face-to-Face Counseling Data Classification and Management System

This study proposes a system for analyzing non-face-to-face counseling data using text-mining techniques to assess psychological states and automatically classify them into predefined categories. The system addresses the challenge of understanding internal issues that may be difficult to express in traditional face-to-face counseling. To solve this problem, a counseling management system based on text mining was developed. In the experiment, we combined TF-IDF and Word Embedding techniques to process and classify client counseling data into five major categories: school, friends, personality, appearance, and family. The classification performance achieved high accuracy and F1-Score, demonstrating the system’s effectiveness in understanding and categorizing clients’ emotions and psychological states. This system offers a structured approach to analyzing counseling data, providing counselors with a foundation for recommending personalized counseling treatments. The findings of this study suggest that in-depth analysis and classification of counseling data can enhance the quality of counseling, even in non-face-to-face environments, offering more efficient and tailored solutions.

Influence maximization under imbalanced heterogeneous networks via lightweight reinforcement learning with prior knowledge

Influence Maximization (IM) stands as a central challenge within the domain of complex network analysis, with the primary objective of identifying an optimal seed set of a predetermined size that maximizes the reach of influence propagation. Over time, numerous methodologies have been proposed to address the IM problem. However, one certain network referred to as Imbalanced Heterogeneous Networks (IHN), which widely used in social situation, urban and rural areas, and merchandising, presents challenges in achieving high-quality solutions. In this work, we introduce the Lightweight Reinforcement Learning algorithm with Prior knowledge (LRLP), which leverages the Struc2Vec graph embedding technique that captures the structural similarity of nodes to generate vector representations for nodes within the network. In details, LRLP incorporates prior knowledge based on a group of centralities, into the initial experience pool, which accelerates the reinforcement learning training for better solutions. Additionally, the node embedding vectors are input into a Deep Q Network (DQN) to commence the lightweight training process. Experimental evaluations conducted on synthetic and real networks showcase the effectiveness of the LRLP algorithm. Notably, the improvement seems to be more pronounced when the the scale of the network is larger. We also analyze the effect of different graph embedding algorithms and prior knowledge on algorithmic results. Moreover, we conduct an analysis about some parameters, such as number of seed set selections T, embedding dimension d and network update frequency C. It is significant that the reduction of number of seed set selections T not only keeps the quality of solutions, but lowers the algorithm’s computational cost.

Dynamics of non-Markovian systems: Markovian embedding versus effective mass approach.

Dynamics of non-Markovian systems is a classic problem yet it attracts everlasting activity in physics and beyond. A powerful tool for modeling such setups is the generalized Langevin equation, however, its analysis typically poses a major challenge even for numerical means. For this reason, various approximations have been proposed over the years that simplify the original model. In this paper, we compare two methods allowing us to tackle this great challenge: (i) the well-known and successful Markovian embedding technique and (ii) the recently developed effective mass approach. We discuss their scope of applicability, numerical accuracy, and computational efficiency. In doing so, we consider a paradigmatic model of a free Brownian particle subjected to power-law correlated thermal noise. We show that when the memory time is short, the effective mass approach offers satisfying precision and typically is much faster than the Markovian embedding. Moreover, the concept of effective mass can be used to find optimal parameters allowing us to reach supreme accuracy and minimal computational cost within the embedding. Our paper therefore provides a blueprint for investigating the dynamics of non-Markovian systems.

The heat modulated infinite dimensional Heston model and its numerical approximation

The HEat modulated Infinite DImensional Heston (HEIDIH) is introduced as a concrete case of the general framework of infinite dimensional Heston stochastic volatility models of (F.E. Benth, I.C. Simonsen '18) for the pricing of forward contracts. It is supported by a comprehensive numerical analysis. The model consists of a one-dimensional stochastic advection equation coupled with a stochastic volatility process. This process is a Cholesky-type decomposition of the tensor product of a Hilbert-space valued Ornstein-Uhlenbeck process, the solution to a stochastic heat equation on the real half-line. The advection and heat equations are driven by independent space-time Gaussian processes which are white in time and coloured in space, with the latter covariance structure expressed by two different kernels. First, a class of weight-stationary kernels are given, under which regularity results for the HEIDIH model in fractional Sobolev spaces are formulated. In particular, the class includes weighted Matérn kernels. Second, numerical approximation of the model is considered. An error decomposition formula, pointwise in space and time, for a finite-difference scheme is proven. For a special case, essentially sharp convergence rates are obtained when this is combined with a fully discrete finite element approximation of the stochastic heat equation. The analysis takes into account a localization error, a pointwise-in-space finite element discretization error and an error stemming from the noise being sampled pointwise in space. The rates obtained in the analysis are higher than what would be obtained using a standard Sobolev embedding technique. Numerical simulations illustrate the results.

Harnessing hybrid deep learning approach for personalized retrieval in e-learning.

The current worldwide pandemic has significantly increased the need for online learning platforms, hence presenting difficulty in choosing appropriate course materials from the vast online educational resources due to user knowledge frameworks variations. This paper presents a novel course recommendation system called the Deep Learning-based Course Recommendation System (DLCRS). The DLCRS combines a hybrid Sequential GRU+adam optimizer with collaborative filtering techniques to offer accurate and learner-centric course suggestions. The proposed approach integrates modules for learner feature extraction and course feature extraction that is performed using (Embeddings from Language Models) ELMO word embedding technique in order to gain a thorough understanding of learner and course profiles and feedback. In order to evaluate the efficacy of the proposed DLCRS, several extensive experiments were carried out utilizing authentic datasets sourced from a reputable public organization. The results indicate a notable area under the receiver operating characteristic curve (AUC) score of 89.62%, which exceeds the performance of similar advanced course recommendation systems. The experimental findings support the viability of the DLCRS, as seen by a significant hit ratio of 0.88, indicating high accuracy in its suggestions.

Unveiling user identity across social media: a novel unsupervised gradient semantic model for accurate and efficient user alignment

The field of social network analysis has identified User Alignment (UA) as a crucial area of investigation. The objective of UA is to identify and connect user accounts across diverse social networks, even when there are no explicit interconnections. UA plays a pivotal role in synthesising coherent user profiles and delving into the intricacies of user behaviour across platforms. However, traditional approaches have encountered limitations. Singular embedding techniques have been found to fall short in fully capturing the semantic essence of user profile attributes. Furthermore, classification-based embedding methods lack definitive criteria for categorisation, thereby constraining both the efficacy and applicability of these models. This paper presents a novel unsupervised Gradient Semantic Model for User Alignment (GSMUA) for the purpose of identifying common user identities across social networks. GSMUA categorises user profile information into weak, sub, and strong gradients based on the semantic intensity of attributes. Different gradient semantic levels direct attention to literal features, semantic features, or a combination of both during feature extraction, thereby achieving a full semantic representation of user attributes. In the case of strongly semantic long texts, GSMUA employs Named Entity Recognition (ENR) technology in order to enhance the inefficient handling of such texts. Furthermore, GSMUA compensates for missing user profile attributes by utilising profile information from user neighbours, thereby reducing the negative impact of missing user profile attributes on model performance. Extensive experiments conducted on four pairs of real datasets demonstrate the superiority of our approach. In comparison to the most effective previously developed unsupervised methods, GSMUA demonstrates improvements in hit-precision ranging from 5.32 to 12.17%. When compared to supervised methods, the improvements range from 0.71 to 11.79%.

A Systematic Literature Review on Automated Software Vulnerability Detection Using Machine Learning

In recent years, numerous Machine Learning (ML) models, including Deep Learning (DL) and classic ML models, have been developed to detect software vulnerabilities. However, there is a notable lack of comprehensive and systematic surveys that summarize, classify, and analyze the applications of these ML models in software vulnerability detection. This absence may lead to critical research areas being overlooked or under-represented, resulting in a skewed understanding of the current state of the art in software vulnerability detection. To close this gap, we propose a comprehensive and systematic literature review that characterizes the different properties of ML-based software vulnerability detection systems using six major Research Questions (RQs). Using a custom web scraper, our systematic approach involves extracting a set of studies from four widely used online digital libraries: ACM Digital Library, IEEE Xplore, ScienceDirect, and Google Scholar. We manually analyzed the extracted studies to filter out irrelevant work unrelated to software vulnerability detection, followed by creating taxonomies and addressing RQs. Our analysis indicates a significant upward trend in applying ML techniques for software vulnerability detection over the past few years, with many studies published in recent years. Prominent conference venues include the International Conference on Software Engineering (ICSE), the International Symposium on Software Reliability Engineering (ISSRE), the Mining Software Repositories (MSR) conference, and the ACM International Conference on the Foundations of Software Engineering (FSE), whereas Information and Software Technology (IST), Computers & Security (C&S), and Journal of Systems and Software (JSS) are the leading journal venues. Our results reveal that 39.1% of the subject studies use hybrid sources, whereas 37.6% of the subject studies utilize benchmark data for software vulnerability detection. Code-based data are the most commonly used data type among subject studies, with source code being the predominant subtype. Graph-based and token-based input representations are the most popular techniques, accounting for 57.2% and 24.6% of the subject studies, respectively. Among the input embedding techniques, graph embedding and token vector embedding are the most frequently used techniques, accounting for 32.6% and 29.7% of the subject studies. Additionally, 88.4% of the subject studies use DL models, with recurrent neural networks and graph neural networks being the most popular subcategories, whereas only 7.2% use classic ML models. Among the vulnerability types covered by the subject studies, CWE-119, CWE-20, and CWE-190 are the most frequent ones. In terms of tools used for software vulnerability detection, Keras with TensorFlow backend and PyTorch libraries are the most frequently used model-building tools, accounting for 42 studies for each. In addition, Joern is the most popular tool used for code representation, accounting for 24 studies. Finally, we summarize the challenges and future directions in the context of software vulnerability detection, providing valuable insights for researchers and practitioners in the field.

Movie Recommendation System Based on Word Embedding

Abstract. Amidst the swift progression of Internet and big data technologies, recommendation systems have emerged as crucial conduits linking users to products across various digital platforms. This study delves into the deployment of neural collaborative filtering within the realm of movie recommendation systems, with the objective of constructing a system of high precision. Utilizing the MovieLens dataset, this investigation applies one-hot encoding and embedding techniques within the PyTorch-Lightning framework to effectively model user behaviors and predict cinematic preferences. The neural collaborative filtering (NCF) model leverages deep learning to extract latent features of users and items, exhibiting a notable enhancement in performance compared to traditional collaborative filtering approaches. Empirical results indicate that the NCF model attains an accuracy of 86% on the validation dataset, underscoring its efficacy in extensive recommendation scenarios.