Neural Embedding Models Research Articles

A Proposed Ensemble Approach for Searching Hacking News Semantically

Efficient search of hacking information has been a topic of great discussion in recent years. Many challenges are encountered when searching for this information. In particular, researchers may encounter unfamiliar and potentially challenging terms, ideas, tools, and other items that are unique to hacking. Effective comprehension of synonyms and polysemy is necessary. These reasons serve as the driving force behind our efforts to develop a productive method for semantic hacking information searches. Semantic search, using advanced NLP techniques, has transformed information retrieval by improving search result accuracy and relevance. Unlike traditional lexical methods, neural models like sentence-transformers handle synonyms and polysemy efficiently. However, processing time increases with model size. This paper proposes a novel ensemble semantic search (NESS) approach that aggregates mini or small neural embedding models, leveraging their distinct advantages. Evaluated on a dataset with over 300,000 Hacker News stories, our proposed method significantly enhances ranking quality and retrieval accuracy compared to existing techniques, while requiring half the processing time of the best-performing large model. The findings underscore the trade-offs between model complexity, retrieval accuracy, and processing efficiency, offering insights for optimizing semantic search systems.

Adversarial regularized autoencoder graph neural network for microbe-disease associations prediction.

Microorganisms inhabit various regions of the human body and significantly contribute to numerous diseases. Predicting the associations between microbes and diseases is crucial for understanding pathogenic mechanisms and informing prevention and treatment strategies. Biological experiments to determine these associations are time-consuming and costly. Therefore, integrating deep learning with biological networks can efficiently identify potential microbe-disease associations on a large scale. We propose an adversarial regularized autoencoder graph neural network algorithm, named Stacked Adversarial Regularization for Microbe-Disease Associations Prediction (SARMDA), for predicting associations between microbes and diseases. First, we integrate topological structural similarity and functional similarity metrics of microbes and diseases to construct a heterogeneous network. Then, utilizing an autoencoder based on GraphSAGE, we learn both the topological and attribute representations of nodes within the constructed network. Finally, we introduce an adversarial regularized autoencoder graph neural network embedding model to address the inherent limitations of traditional GraphSAGE autoencoders in capturing global information. Under the five-fold cross-validation on microbe-disease pairs, SARMDA was compared with eight advanced methods using the Human Microbe-Disease Association Database (HMDAD) and Disbiome databases. The best area under the ROC curve (AUC) achieved by SARMDA on HMDAD was 0.9891$\pm$0.0057, and the best area under the precision-recall curve (AUPR) was 0.9902$\pm$0.0128. On the Disbiome dataset, the AUC was 0.9328$\pm$0.0072, and the best AUPR was 0.9233$\pm$0.0089, outperforming the other eight MDAs prediction methods. Furthermore, the effectiveness of our model was demonstrated through a detailed analysis of asthma and inflammatory bowel disease cases.

Automatically Temporal Labeled Data Generation Using Positional Lexicon Expansion for Focus Time Estimation of News Articles

Many facts change over time, which is a fundamental aspect of our physical environment. In the case of pandemic articles, the user is not interested in the creation date of the document but in the facts and the cause of the last pandemic. Fake news can be better combated by having a document with a temporal focus. Currently, neither the sequence of events nor the temporal focus is considered when obtaining news documents. Despite the limited number of temporal aspects in the available datasets, it is difficult to test and evaluate the temporal conclusions of the model. The goal of this work is to develop a temporal focus news article retrieval model based on co-training to advance research in semi-supervised learning. A mapping of the dataset is performed using (1) the evolving focus time of news articles and (2) the semi-supervised method based on coincidence contexts for learning low-dimensional continuous vectors for learning neural contrast embedding models generating focus time-based query in sequential news articles to facilitate temporal understanding by learning low-dimensional continuous vectors. A diverse dataset of news articles is used to evaluate the effectiveness of the proposed method. With semi-supervised learning and lexicon expansion, the result of the developed model can achieve 89%. The method performed better than previous baselines and traditional machine learning models with improvements of 12.65% and 4.7%, respectively.

ConvTKG: A query-aware convolutional neural network-based embedding model for temporal knowledge graph completion

Temporal knowledge graphs (TKGs) are still far from completeness although they have benefited many artificial intelligence applications successfully. To alleviate the incompleteness of TKGs, temporal knowledge graph completion (TKGC) aims to perform link prediction and reason new time-sensitive facts from existing facts. Among the various TKGC methods, temporal knowledge graph embedding (TKGE) methods represent entities, relations, and timestamps in low-dimensional spaces, which are known for their high reasoning performance and robust scalability. Some previous researches on TKGE aim to endow the embedding quadruple with the translational characteristic by modeling the global relationship in the embedding quadruple. However, the weights of same dimensional entries of the embedding quadruple are fixed and unique, which limits the ability to capture diverse global relationships. Besides, most TKGE models view timestamps as a kind of general feature and overlook their interrelations and correlations with the queries. Moreover, the distinct characteristics of an entity as the subject entity and the object entity of facts have not been fully leveraged. In this paper, we propose a query-aware embedding model based on the convolutional neural network (CNN) named ConvTKG to perform TKGE, thus tackling the TKGC task. We design a new temporal information encoder based on the Gated Recurrent Unit (GRU) and attention mechanism to learn the query-aware representation of temporal information. More importantly, we utilize a CNN-based decoder where multiple one-dimensional convolution kernels are operated on the matrix composed of entity embedding, relation embedding, and temporal representation to capture global relationships among them. In addition, we assign two independent vectors for each entity and take advantage of the inverse relation to allow them to be learned dependently. Experiments show that ConvTKG achieves better link prediction performance than previous state-of-the-art baseline methods for TKGC on three benchmark datasets: ICEWS14, ICEWS05-15, and GDELT.

Multi-Filter soft shrinkage network for knowledge graph embedding

Incompleteness is a prominent issue pervasive in real-world knowledge graphs, and link prediction techniques, which utilize known facts to forecast missing or unknown links, have emerged as a critical technology for knowledge graph completion. Recently, convolutional neural network-based knowledge graph embedding (KGE) models have achieved breakthroughs in the field of link prediction by increasing the number of interactions between entities and relations. However, current models mainly emphasize stacking complex architectures to capture more interaction features, ignoring the fact that, as the number of features increases significantly, irrelevant features might overshadow important ones, thus impacting the performance of link prediction. To tackle this problem, this paper introduces a novel KGE model that is based on a multi-filter soft shrinkage network (MFSSN). First, this model adaptively constructs filters based on entity embedding and relationship embedding, thereby increasing the interactions between entities and relationships. Secondly, the soft shrinkage function is innovatively introduced into the model, and a specialized soft shrinkage sub-network is innovatively designed to effectively eliminate noise features and improve attention to important features. Finally, the attention mechanism is added to the network to enhance important features and suppress useless features, improving the ability to utilize features, and thereby enhancing model performance. Through extensive experiments on five benchmark datasets of different sizes, the strong performance and generalization ability of the proposed model are proven. Compared with contrasting methods, it achieves performance leadership in almost all indicators.

Neural embeddings of urban big data reveal spatial structures in cities

Over decades, many cities have been expanded and functionally diversified by population activities, socio-demographics and attributes of the built environment. Urban expansion and development have led to the emergence of spatial structures of cities. Uncovering cities’ spatial structures is critical to understanding various urban phenomena such as segregation, equity of access, and sustainability. In this study, we propose using a neural embedding model—graph neural network (GNN)—that leverages the heterogeneous features of urban areas and their interactions captured by human mobility networks to obtain vector representations of these areas. Using large-scale high-resolution mobility data sets from millions of aggregated and anonymized mobile phone users in 16 metropolitan counties in the United States, we demonstrate that our embeddings encode complex relationships among features related to urban components (such as distribution of facilities) and population attributes and activities. The clustered representations of urban areas show the shared characteristics among urban areas in the same cluster. We show that embeddings generated by a model trained on a different county can capture 50% to 60% of the spatial structure in another county, allowing us to make cross-county comparisons and inferences. The findings reveal complex relationships among urban components in cities. Since the identified multifaceted spatial structures capture the combined effects of various mechanisms, such as segregation, disparate facility distribution, and human mobility, the findings could help identify the limitations of the current city structure to inform planning decisions and policies. Also, the model and findings set the stage for a variety of research in urban planning, engineering and social science through an integrated understanding of how the complex interactions between urban components and population activities and attributes shape the spatial structures in cities.

Neural embeddings of scientific mobility reveal the stratification of institutions in China

We are trying to reveal the status of Chinese universities in the global talent circulation system and how university prestige affects mobility patterns in China. Other than geographic distance, we configured a two-dimensional Euclidean space of the neural embeddings of institutions from scientists’ mobility trajectories. To be specific, we acquired the scientific mobility trajectories based on about 3,055,409 authors from 108 thousand affiliations and used the word2vec neural embedding model to encode affiliations into a continuous 100-dimensional embedding space and then further map the 55,298 distinct embeddings into the two-dimensional space. We found that in the current talent circulation system, China is furthest from the scientific center with a closeness of 0.08 and has a median diversity of 0.15 among the top 10 countries with the most affiliations, indicating relatively less communication with other international institutions and moderate diversity of communication patterns within the country. Furthermore, using the regression models, we found that for the universities on China's mainland, the effects of prestige are positive on the volumes of scientific mobility (all pvalues<0.001) and on the uniqueness of universities (all pvalues<0.1), revealing the stratifications of universities in the talent circulation. The interaction of prestige and regional economic status is also discussed in our results. This work gives policy implications to university development, knowledge circulation, and career development in science.

Unsupervised embedding of trajectories captures the latent structure of scientific migration.

Human migration and mobility drives major societal phenomena including epidemics, economies, innovation, and the diffusion of ideas. Although human mobility and migration have been heavily constrained by geographic distance throughout the history, advances, and globalization are making other factors such as language and culture increasingly more important. Advances in neural embedding models, originally designed for natural language, provide an opportunity to tame this complexity and open new avenues for the study of migration. Here, we demonstrate the ability of the model word2vec to encode nuanced relationships between discrete locations from migration trajectories, producing an accurate, dense, continuous, and meaningful vector-space representation. The resulting representation provides a functional distance between locations, as well as a "digital double" that can be distributed, re-used, and itself interrogated to understand the many dimensions of migration. We show that the unique power of word2vec to encode migration patterns stems from its mathematical equivalence with the gravity model of mobility. Focusing on the case of scientific migration, we apply word2vec to a database of three million migration trajectories of scientists derived from the affiliations listed on their publication records. Using techniques that leverage its semantic structure, we demonstrate that embeddings can learn the rich structure that underpins scientific migration, such as cultural, linguistic, and prestige relationships at multiple levels of granularity. Our results provide a theoretical foundation and methodological framework for using neural embeddings to represent and understand migration both within and beyond science.

Characterizing Designs Via Isometric Embeddings: Applications to Airfoil Inverse Design

Abstract Many design problems involve reasoning about points in high-dimensional space. A common strategy is to first embed these high-dimensional points into a low-dimensional latent space. We propose that a good embedding should be isometric—i.e., preserving the geodesic distance between points on the data manifold in the latent space. However, enforcing isometry is non-trivial for common neural embedding models such as autoencoders. Moreover, while theoretically appealing, it is unclear to what extent is enforcing isometry necessary for a given design analysis. This paper answers these questions by constructing an isometric embedding via an isometric autoencoder, which we employ to analyze an inverse airfoil design problem. Specifically, the paper describes how to train an isometric autoencoder and demonstrates its usefulness compared to non-isometric autoencoders on the UIUC airfoil dataset. Our ablation study illustrates that enforcing isometry is necessary for accurately discovering clusters through the latent space. We also show how isometric autoencoders can uncover pathologies in typical gradient-based shape optimization solvers through an analysis on the SU2-optimized airfoil dataset, wherein we find an over-reliance of the gradient solver on the angle of attack. Overall, this paper motivates the use of isometry constraints in neural embedding models, particularly in cases where researchers or designers intend to use distance-based analysis measures to analyze designs within the latent space. While this work focuses on airfoil design as an illustrative example, it applies to any domain where analyzing isometric design or data embeddings would be useful.

HiPrompt: Few-Shot Biomedical Knowledge Fusion via Hierarchy-Oriented Prompting.

Medical decision-making processes can be enhanced by comprehensive biomedical knowledge bases, which require fusing knowledge graphs constructed from different sources via a uniform index system. The index system often organizes biomedical terms in a hierarchy to provide the aligned entities with fine-grained granularity. To address the challenge of scarce supervision in the biomedical knowledge fusion (BKF) task, researchers have proposed various unsupervised methods. However, these methods heavily rely on ad-hoc lexical and structural matching algorithms, which fail to capture the rich semantics conveyed by biomedical entities and terms. Recently, neural embedding models have proved effective in semantic-rich tasks, but they rely on sufficient labeled data to be adequately trained. To bridge the gap between the scarce-labeled BKF and neural embedding models, we propose HiPrompt, a supervision-efficient knowledge fusion framework that elicits the few-shot reasoning ability of large language models through hierarchy-oriented prompts. Empirical results on the collected KG-Hi-BKF benchmark datasets demonstrate the effectiveness of HiPrompt.

Neural network based integration of assays to assess pathogenic potential

Limited data significantly hinders our capability of biothreat assessment of novel bacterial strains. Integration of data from additional sources that can provide context about the strain can address this challenge. Datasets from different sources, however, are generated with a specific objective and which makes integration challenging. Here, we developed a deep learning-based approach called the neural network embedding model (NNEM) that integrates data from conventional assays designed to classify species with new assays that interrogate hallmarks of pathogenicity for biothreat assessment. We used a dataset of metabolic characteristics from a de-identified set of known bacterial strains that the Special Bacteriology Reference Laboratory (SBRL) of the Centers for Disease Control and Prevention (CDC) has curated for use in species identification. The NNEM transformed results from SBRL assays into vectors to supplement unrelated pathogenicity assays from de-identified microbes. The enrichment resulted in a significant improvement in accuracy of 9% for biothreat. Importantly, the dataset used in our analysis is large, but noisy. Therefore, the performance of our system is expected to improve as additional types of pathogenicity assays are developed and deployed. The proposed NNEM strategy thus provides a generalizable framework for enrichment of datasets with previously collected assays indicative of species.

Neural age screening on question answering communities

For online social networks, demographic analysis is absolutely essential for improving their services in many ways. It is instrumental in understanding their different audiences, members and competitors. As well as that, it is pivotal in designing effective personalization and contextualization strategies, especially for displaying and creating better content. There is, for this reason, a great bulk of research into how demographic variables are characterized and how they impact online platforms such as Facebook and Twitter. But surprisingly, only a handful of works delve into their characterization and effect on community Question-Answering (cQA) websites. In this particular context, the subject of age demographics remains largely unexplored.This paper takes the lead on interpreting automatic age recognition on CQAs (a.k.a. age screening) as a regression task. To this effect, it compares state-of-the-art graph-based neural network regression and embedding models on a massive activity-graph encompassing ca. 16 and 837 million nodes (members) and edges, respectively. For this study, a large-scale subset of ca. 657,000 community fellows was automatically associated with their age via aligning their profile texts with a limited number of linguistic patterns.In short, our results show that Node2vec significantly outperforms other embeddings regardless of the regression model used for casting predictions. When this embedding is combined with Artificial Neural Network Regressions, we obtained our best configuration scoring a Root Mean Square Error (RMSE) of 8.39. An interesting qualitative feature of this embedding space is that age-based centroid vectors tend to form a trail ordered by age. Lastly, our outcomes also signal that activity graph based models can rival its counterparts based on image and textual inputs, paving the way for constructing effective multi-modal approaches.

ON THE EFFECTIVENESS OF PARAGRAPH VECTOR MODELS IN DOCUMENT SIMILARITY ESTIMATION FOR TURKISH NEWS CATEGORIZATION

News categorization, which is a common application area of text classification, is the task of automatic annotation of news articles with predefined categories. In parallel with the rise of deep learning techniques in the field of machine learning, neural embedding models have been widely utilized to capture hidden relationships and similarities among textual representations of news articles. In this study, we approach the Turkish news categorization problem as an ad-hoc retrieval task and investigate the effectiveness of paragraph vector models to compute and utilize document-wise similarities of Turkish news articles. We propose an ensemble categorization approach that consists of three main stages, namely, document processing, paragraph vector learning, and document similarity estimation. Extensive experiments conducted on the TTC-3600 dataset reveal that the proposed system can reach up to 93.5% classification accuracy, which is a remarkable performance when compared to the baseline and state-of-the-art methods. Moreover, it is also shown that the Distributed Bag of Words version of Paragraph Vectors performs better than the Distributed Memory Model of Paragraph Vectors in terms of both accuracy and computational performance.

HNTSumm: Hybrid text summarization of transliterated news articles

Data generated from social networking sites, blogs, digital magazines, and news websites is the largest human-generated data. Summarization is the process of extracting the crux of a document which when done manually can be tedious and deluging. Automatic text summarization is an approach that encapsulates long documents into a few sentences or words by enwrapping the gist and the principal information of the document. With the growth of social networking sites, eBooks, and e-Papers, the prevalence of transliterated words in text corpora is also on the rise. In this paper, we propose a word embeddings-based algorithm called HNTSumm by combining the advantages of unsupervised and supervised learning methods. The proposed algorithm HNTSumm algorithm is an imminent method for automatic text summarization of huge volumes of data that can learn word embeddings for words transliterated from other languages to English by utilizing weighted word embeddings from a Neural Embedding Model. Further, the amalgamation of extractive and abstractive approaches yields a concise and unambiguous summary of the text documents as the extractive approach eliminates redundant information. We employ a hybrid version of the Sequence-to-sequence models to generate an abstractive summary for the transliterated words. The feasibility of this algorithm was evaluated using two different news summary datasets and the accuracy scores were computed with the ROUGE evaluation metric. Experimental results corroborate the higher performance of the proposed algorithm and show HNTSumm outperforms relevant state-of-the-art algorithms for datasets with transliterated words.

Temporal positional lexicon expansion for federated learning based on hyperpatism detection

AbstractInternet‐based information exchange has resulted in the propagation of false and misleading information, which is highly detrimental to individuals and humankind. Due to the speed and volume of social media news production, supervised artificial intelligence algorithms require many annotated data, which is difficult, costly, and time‐consuming. To address this issue, we offer a novel federated semi‐supervised framework based on self‐ensembling that utilizes the linguistic and stylometric information of annotated news articles and searches for hidden patterns in unlabeled data to denoise labels. Self‐ensembling predicts the labels of unlabeled data by using the outcomes of network‐in‐training from earlier epochs. These cumulative predictions should be a stronger predictor for unknown labels than the output of the most recent training epoch; hence, they may be utilized as a substitute for the labels of unlabeled data. The approach is distinctive in collecting all of the outputs from the neural network's past training periods. It utilizes them as an unsupervised target against which to assess the current output prediction of unlabeled articles. We intend to create a dataset centred on denoising to forward the study. The dataset is mapped using (1) the shifting focus time from published news articles and (2) the semi‐supervised method based on coincidence contexts for a neural contrast embedding model for learning low‐dimensional continuous vectors that generate a focus time‐based query in sequential news articles for temporal comprehension. The model achieved 0.83% F‐measure with lexicon expansion semi‐supervised learning.

Autoencoder-convolutional neural network-based embedding and extraction model for image watermarking

Watermarking consists of embedding in, and later extracting from, a digital cover a design called a watermark to prove the image’s copyright/ownership. In watermarking, the use of deep-learning approaches is extremely beneficial due to their strong learning ability with accurate and superior results. By taking advantage of deep-learning, we designed an autoencoder convolutional neural network (CNN)-based watermarking algorithm to maximize the robustness while ensuring the invisibility of the watermark. A two network model, including embedding and extraction, is introduced to comprehensively analyze the performance of the algorithm. The embedding network architecture is composed of convolutional autoencoders. Initially, CNN is considered to obtain the feature maps from the cover and mark images. Subsequently, the feature maps of the mark and cover are concatenated with the help of the concatenation principle. In the extraction model, block-level transposed convolution and the rectified linear unit algorithm is applied on the extracted features of watermarked and cover images to obtain the hidden mark. Extensive experiments demonstrate that the proposed algorithm has high invisibility and good robustness against several attacks at a low cost. Further, our proposed scheme outperforms other state-of-the-art schemes in terms of robustness with good invisibility.

Learning knowledge graph embeddings by deep relational roto-reflection

Embedding methods map entities and relations to low-dimensional vectors and then use a scoring function to predict missing links for knowledge graph completion. Most of the existing deep embedding methods are primarily based on planner convolution, which generates less expressive feature maps that limit the predictive performance. In this paper, we propose a group convolution and hypernetwork-based neural embedding model named, DeepER for knowledge graph completion. DeepER utilizes rotation and reflection transformations of group convolution to produce more expressive feature maps for entities and relations. Furthermore, it introduces a relation-specific roto-reflection of head entities via hypernetwork architecture to preserve the relation-specific information in the embeddings while keeping the rotation and reflection properties of the relations. In addition, we introduce a multimodal extension of DeepER that includes visual and structured information in the embedding vectors of entities. Experimental results demonstrate that DeepER outperforms 20 existing methods on knowledge graph completion benchmarks, consisting of both structured and multimodal datasets.

An Enhanced Neural Word Embedding Model for Transfer Learning

Due to the expansion of data generation, more and more natural language processing (NLP) tasks are needing to be solved. For this, word representation plays a vital role. Computation-based word embedding in various high languages is very useful. However, until now, low-resource languages such as Bangla have had very limited resources available in terms of models, toolkits, and datasets. Considering this fact, in this paper, an enhanced BanglaFastText word embedding model is developed using Python and two large pre-trained Bangla models of FastText (Skip-gram and cbow). These pre-trained models were trained on a collected large Bangla corpus (around 20 million points of text data, in which every paragraph of text is considered as a data point). BanglaFastText outperformed Facebook’s FastText by a significant margin. To evaluate and analyze the performance of these pre-trained models, the proposed work accomplished text classification based on three popular textual Bangla datasets, and developed models using various machine learning classical approaches, as well as a deep neural network. The evaluations showed a superior performance over existing word embedding techniques and the Facebook Bangla FastText pre-trained model for Bangla NLP. In addition, the performance in the original work concerning these textual datasets provides excellent results. A Python toolkit is proposed, which is convenient for accessing the models and using the models for word embedding, obtaining semantic relationships word-by-word or sentence-by-sentence; sentence embedding for classical machine learning approaches; and also the unsupervised finetuning of any Bangla linguistic dataset.

SAAED: Embedding and Deep Learning Enhance Accurate Prediction of Association Between circRNA and Disease.

Emerging evidence indicates that circRNA can regulate various diseases. However, the mechanisms of circRNA in these diseases have not been fully understood. Therefore, detecting potential circRNA–disease associations has far-reaching significance for pathological development and treatment of these diseases. In recent years, deep learning models are used in association analysis of circRNA–disease, but a lack of circRNA–disease association data limits further improvement. Therefore, there is an urgent need to mine more semantic information from data. In this paper, we propose a novel method called Semantic Association Analysis by Embedding and Deep learning (SAAED), which consists of two parts, a neural network embedding model called Entity Relation Network (ERN) and a Pseudo-Siamese network (PSN) for analysis. ERN can fuse multiple sources of data and express the information with low-dimensional embedding vectors. PSN can extract the feature between circRNA and disease for the association analysis. CircRNA–disease, circRNA–miRNA, disease–gene, disease–miRNA, disease–lncRNA, and disease–drug association information are used in this paper. More association data can be introduced for analysis without restriction. Based on the CircR2Disease benchmark dataset for evaluation, a fivefold cross-validation experiment showed an AUC of 98.92%, an accuracy of 95.39%, and a sensitivity of 93.06%. Compared with other state-of-the-art models, SAAED achieves the best overall performance. SAAED can expand the expression of the biological related information and is an efficient method for predicting potential circRNA–disease association.

Lazy Network: A Word Embedding‐Based Temporal Financial Network to Avoid Economic Shocks in Asset Pricing Models

Public companies in the US stock market must annually report their activities and financial performances to the SEC by filing the so‐called 10‐K form. Recent studies have demonstrated that changes in the textual content of the corporate annual filing (10‐K) can convey strong signals of companies’ future returns. In this study, we combine natural language processing techniques and network science to introduce a novel 10‐K‐based network, named Lazy Network, that leverages year‐on‐year changes in companies’ 10‐Ks detected using a neural network embedding model. The Lazy Network aims to capture textual changes derived from financial or economic changes on the equity market. Leveraging the Lazy Network, we present a novel investment strategy that attempts to select the least disrupted and stable companies by capturing the peripheries of the Lazy Network. We show that this strategy earns statistically significant risk‐adjusted excess returns. Specifically, the proposed portfolios yield up to 95 basis points in monthly five‐factor alphas (over 12% annually), outperforming similar strategies in the literature.