Embedding Quality Research Articles

Embedding quality in primary healthcare

Embedding quality in primary healthcare

One-shot Based Knowledge Graph Embedded Neural Architecture Search Algorithm

The quality of embeddings is crucial for downstream tasks in knowledge graphs. Researchers usually introduce neural network architecture search into knowledge graph embedding for machine automatic construction of appropriate neural networks for each dataset. An existing approach is to divide the search space into macro search space and micro search space. The search strategy for micro space is based on one-shot weight sharing strategy, but it will lead to all the information obtained from the previous supernet training is discarded and the advantages of one-shot algorithm are not fully utilized. In this paper, we conduct experiments on common datasets for two important downstream tasks of knowledge graph embedding entity alignment and link prediction problems, respectively-and compare the search performance with existing manually designed neural networks as well as good neural network search algorithms. The results show that the improved algorithm can search better architectures for the same time when experiments are performed on the same dataset; the improved algorithm takes less time to search architectures with similar performance. Also, the improved algorithm searched the model on the dataset due to the human optimal level.

Heart disease risk factors detection from electronic health records using advanced NLP and deep learning techniques

Heart disease remains the major cause of death, despite recent improvements in prediction and prevention. Risk factor identification is the main step in diagnosing and preventing heart disease. Automatically detecting risk factors for heart disease in clinical notes can help with disease progression modeling and clinical decision-making. Many studies have attempted to detect risk factors for heart disease, but none have identified all risk factors. These studies have proposed hybrid systems that combine knowledge-driven and data-driven techniques, based on dictionaries, rules, and machine learning methods that require significant human effort. The National Center for Informatics for Integrating Biology and Beyond (i2b2) proposed a clinical natural language processing (NLP) challenge in 2014, with a track (track2) focused on detecting risk factors for heart disease risk factors in clinical notes over time. Clinical narratives provide a wealth of information that can be extracted using NLP and Deep Learning techniques. The objective of this paper is to improve on previous work in this area as part of the 2014 i2b2 challenge by identifying tags and attributes relevant to disease diagnosis, risk factors, and medications by providing advanced techniques of using stacked word embeddings. The i2b2 heart disease risk factors challenge dataset has shown significant improvement by using the approach of stacking embeddings, which combines various embeddings. Our model achieved an F1 score of 93.66% by using BERT and character embeddings (CHARACTER-BERT Embedding) stacking. The proposed model has significant results compared to all other models and systems that we developed for the 2014 i2b2 challenge.

Fusing Attribute Character Embeddings with Truncated Negative Sampling for Entity Alignment

Finding pairs of entities from two different knowledge graphs that reflect the same real-world object is the purpose of entity alignment for knowledge graphs. In recent years, techniques that use entity alignment for knowledge fusion have received widespread attention. In this article, we suggest a method for entity alignment using truncated negative sampling with attribute character embedding. The method mainly makes use of the relationship and attribute data in heterogeneous knowledge graphs to fulfil the entity alignment task. Firstly, the framework uses relationship mapping to unify the namespace of heterogeneous relationships. Secondly, the attribute character embeddings are generated using the attribute triples in the knowledge graph to unify the embedding space of heterogeneous entities. Then, the entity similarity between heterogeneous knowledge graphs is captured by structural embedding. Next, to learn more useful semantic information during negative sampling, the framework adopts a truncated negative sampling strategy to increase the generalizability of the model. The negative sampling procedure employs targets with high similarity to the target entity as negative sample targets. Finally, we performed comparison tests on two well-known real-world datasets, and the outcomes demonstrate that the proposed model outperforms three other representative advanced approaches, especially with an over 10% improvement in the Hits@k metric compared to the baseline method.

Automatic ICD-10 coding: Deep semantic matching based on analogical reasoning

BackgroundICD-10 has been widely used in statistical analysis of mortality rates and medical reimbursement. Automatic ICD-10 coding is desperately needed because manually assigning codes is expensive, time-consuming, and labor-intensive. Diagnoses described in medical records differ significantly from those used in ICD-10 classification, making it impossible for existing automatic coding techniques to perform well enough to support medical billing, resource allocation, and research requirements. Meanwhile, most of the current automatic coding approaches are oriented toward English ICD-10. This method for automatically assigning ICD-10 codes to diagnoses extracted from Chinese discharge records was provided in this paper. MethodFirst, BERT creates word representations of the two texts. Second, the context representation layer incorporates contextual information into the representation of each time step of the word representations using a bidirectional Long Short-Term Memory. Third, the matching layer compares each contextual embedding of the uncoded diagnosis record against a weighted version of all contextual character embeddings of the manually coded diagnosis record. The matching strategy is element-wise subtraction and element-wise multiplication and then through a neural network layer. Fourth, the matching vectors are combined using a one-layer convolutional neural network. A sigmoid is then used to output matching results. ResultsTo evaluate the proposed method, 1,003,558 manually coded primary diagnoses were gathered from the homepage of the discharge medical records. The experimental results showed that the proposed method outperformed popular deep semantic matching algorithms, such as DSSM, ConvNet, ESIM, and ABCNN, and demonstrated state-of-the-art results in a single text matching with an accuracy of 0.986, a precision of 0.979, a recall of 0.983, and an F1-score of 0.981. ConclusionThe automatic ICD-10 coding of Chinese diagnoses is successful when using the proposed deep semantic matching approach based on analogical reasoning.

Joint multi-view character embedding model for named entity recognition of Chinese car reviews

Joint multi-view character embedding model for named entity recognition of Chinese car reviews

Loanword identification based on web resources: A case study on wikipedia

To alleviate the resource scarcity and improve the robustness in loanword identification, the current study proposes a novel loanword identification method based on Wikipedia. In this paper, we first present how to obtain loanword candidate datasets and comparable corpora from Wikipedia. On the basis of these corpora, we develop a pseudo-data generation model for loanword identification tasks. And then we put forward a loanword identification model, i.e. the PK-SM-Bi-LSTM-CRF framework, which is based on a bidirectional LSTM-CRF framework and further enhanced by prior knowledge and self-matching attention. The advantages of the proposed method mainly lie in two aspects. For one thing, besides the commonly used word embedding and character embedding features, several other features, including subword embedding, lexical similarity, word alignment and semantic similarity, are incorporated into our method. For another, geographic distance is set as a primary principle in the selection of the best matched donor word from several candidates. Moreover, in order to evaluate the effectiveness of the proposed method, we have conducted a series of experiments in different languages. Experimental results show that the proposed method achieves the best performance among all baseline systems.

Dynamic heterogeneous graph representation learning with neighborhood type modeling

Graph representation learning aims to learn the representations of graph structured data in low-dimensional space, and has a wide range of applications in graph analysis tasks. Real-world networks are generally heterogeneous and dynamic, which contain multiple types of nodes and edges, and the graph may evolve at a high speed over time. The complex heterogeneous properties and rapidly evolving graph structures make it difficult to learn high-quality graph representations for dynamic heterogeneous graphs. Currently, studies concentrated on representation learning of temporal heterogeneous networks are insufficient. Existing methods either rely on meta-paths where the embedding quality heavily depending on experts’ selection, or use network snapshots where the fine-grained temporal information cannot be captured. In this paper, we propose a novel graph neural network model–node signature based Temporal Heterogeneous Graph Attention Network, termed as THGAT, for learning the representations of dynamic heterogeneous networks. THGAT improves the aggregation way of neighborhood information, and pays attention to the enlightenment of the importance of neighbor nodes by heterogeneous information and temporal information that cannot be ignored in the network. We also innovatively propose three node signature methods for encoding the heterogeneous information of the nodes and use the time encoding technique suitable for real-time networks to directly represent the temporal information, so as to overcome the limitations of existing methods. We conduct experiments on four real-world datasets, and the results demonstrate that THGAT improves the representation learning quality significantly, in aspects of link prediction, node classification, and node clustering, compared to the state-of-the-art methods. To make the work more complete, we also analyze the applicable scenarios of the three node signature methods through experiments, respectively.

Adversarial Spam Detector With Character Similarity Network

More and more social platforms suffer from the spam messages and trouble in detecting them. Lizhi, one of the most famous audio APPs in China, also suffers from the spam messages. The spam detector in Lizhi faces two major challenges: adversarial actions taken by the spammers and lack of labeled data. In this article, we propose a novel adversarial spam detector based on character similarity network for detecting adversarial spam messages in Lizhi, namely Lizhi adversarial spam detector. The character similarity network is designed to solve the adversarial actions before being taken by the spammers, and the character embedding model is proposed to learn the embeddings of all the characters in the corpus. Then, the model generates the sentence embedding of the messages sent by users. At last the classifier will predict whether the message is a spam. Also, the model utilizes active learning to solve the problem of lack of labeled data. Both of offline and online experiments are conducted to confirm the effectiveness of the proposed method.

MEQA: Manifold embedding quality assessment via anisotropic scaling and Kolmogorov-Smirnov test

Manifold learning methods unfold the manifold structures and embed them in a lower-dimensional space. The quality of such an embedding should be measured both qualitatively and quantitatively. The proposed manifold embedding quality assessment (MEQA) method does so by taking into account of local and global structure preservation as both are important traits of an embedding. To measure the local structure preservation MEQA uses two transformations. Initially anisotropic scaling, rotation and translation are incorporated to measure the closeness between the original and the embedded data points. In the next stage, rigid transformation is incorporated to quantify the previous transformation which involved anisotropic scaling. For quantifying the global structure preservation, the Kolmogorov-Smirnov test is applied in a distributed manner over each dimension and averaged over them. To establish the superiority of MEQA we conducted several studies over standard synthetic and real-life datasets across separate existing feature extraction techniques.

A Multi-Granularity Word Fusion Method for Chinese NER

Named entity recognition (NER) plays a crucial role in many downstream natural language processing (NLP) tasks. It is challenging for Chinese NER because of certain features of Chinese. Recently, large-scaled pre-training language models have been used in Chinese NER. However, since some of the pre-training language models do not use word information or just employ word information of single granularity, the semantic information in sentences could not be fully captured, which affects these models’ performance. To fully take advantage of word information and obtain richer semantic information, we propose a multi-granularity word fusion method for Chinese NER. We introduce multi-granularity word information into our model. To make full use of the information, we classify the information into three kinds: strong information, moderate information, and weak information. These kinds of information are encoded by encoders and then integrated with each other through the strong-weak feedback attention mechanism. Specifically, we apply two separate attention networks to word embeddings and N-grams embeddings. Then, the outputs are fused into another attention. In these three attentions, character embeddings are used to be the query of attentions. We call the results the multi-granularity word information. To combine character information and multi-granularity word information, we introduce two fusion strategies for better performance. The process makes our model obtain rich semantic information and reduces word segmentation errors and noise in an explicit way. We design experiments to get our model’s best performance by comparing some components. Ablation study is used to verify the effectiveness of each module. The final experiments are conducted on four Chinese NER benchmark datasets and the F1 scores are 81.51% for Ontonotes4.0, 95.47% for MSRA, 95.87% for Resume, and 69.41% for Weibo. The best improvement achieved by the proposed method is 1.37%. Experimental results show that our method outperforms most baselines and achieves the state-of-the-art method in performance.

A Study on the Performance of Recurrent Neural Network based Models in Maithili Part of Speech Tagging

This article presents our effort in developing a Maithili Part of Speech (POS) tagger. Substantial effort has been devoted to developing POS taggers in several Indian languages, including Hindi, Bengali, Tamil, Telugu, Kannada, Punjabi, and Marathi; but Maithili did not achieve much attention from the research community. Maithili is one of the official languages of India, with around 50 million native speakers. So, we worked on developing a POS tagger in Maithili. For the development, we use a manually annotated in-house Maithili corpus containing 56,126 tokens. The tagset contains 27 tags. We train a conditional random fields (CRF) classifier to prepare a baseline system that achieves an accuracy of 82.67%. Then, we employ several recurrent neural networks (RNN)-based models, including Long-short Term Memory (LSTM), Gated Recurrent Unit (GRU), LSTM with a CRF layer (LSTM-CRF), and GRU with a CRF layer (GRU-CRF) and perform a comparative study. We also study the effect of both word embedding and character embedding in the task. The highest accuracy of the system is 91.53%.

Successful learning with whiteboard animations – A question of their procedural character or narrative embedding?

Although whiteboard animations are increasingly used for educational purposes, there is little empirical evidence as to why such animations can enhance learning. To specify essential elements, their dynamic visual presentation, as well as their narrative embedding, were found to be theortically important. In a first Experiment (N = 133) with a 2 (presentation mode: static pictures vs. progressive drawing) x 2 (narrative context: with vs. without a narrative) between-subject factorial design, motivational, cognitive, affective variables, as well as learning outcomes, of secondary school students were measured. Results revealed that progressive drawing, as well as a narrative context, are mostly associated with an increase in learning-relevant variables. In a second experiment with the same sample and the same experimental design but a different whiteboard animation, results from Experiment 1 generalize to another learning content. Again, a progressive drawing, as well as a narrative context within whiteboard animation, fostered learning relevant variables as well as learning outcomes. Results are discussed considering the cognitive theory of multimedia learning, the contiguity effect as well as the instructional design theory of anchored instruction.

Deep learning based phishing website identification system using CNN-LSTM classifier

The term phishing refers to an attack that pretends to be the website of a large corporation, typically one dealing with money, such as a bank or other financial institution or an online retailer. Its primary objective is to acquire personally identifiable information from users, such as their social security numbers, credit card information, and passwords. Due to the rise of phishing attacks, various techniques have been developed in order to combat these threats. One of these is deep learning algorithms, which are capable of learning and analyzing massive datasets. Due to their capabilities, these algorithms are very useful in identifying and preventing phishing attacks. Due to the complexity of the phishing websites, many development systems have been created to detect them. Unfortunately, the output that was desired cannot be achieved by these systems, and they have a number of other flaws as well. The purpose of this paper is to propose a hybrid deep learning-based phishing detection system that is easy to put into practice. The quality of the input dataset is improved through the process of preprocessing the dataset. After that, the procedures of clustering and feature selection are carried out in order to improve the accuracy and decrease the amount of time required for the processing. The resulting features are then fed into the CNN_LSTM, which is a classification system that classifies websites that are phishing and legitimate. Proposed Hybrid deep learning models are proposed to combine the features of natural language processing (NLP) and character embedding. They can then reveal high-level connections between characters. In terms of the metric that is being used for the evaluation, the performance of the models that have been proposed is better than that of the other models.

Improving ANiTW Performance Using Bigrams Character Encoding and Identity-Based Signature

Several threats, such as misinformation and hoax, exist in social media as a frequently used technology to exchange information. These threats can be prevented if the reader can verify the message’s content integrity and source authenticity. One recent solution to overcome these threats was proposed by Ahvanooey et al. called ANiTW. ANiTW uses an invisible text watermarking scheme to verify social media message integrity and identify the message source. However, ANiTW is still vulnerable to undetected malicious text modification. Moreover, ANiTW has no tools to authenticate the extracted message source. The proposed method aims to overcome those vulnerabilities and improve ANiTW performance as a tool to verify message content integrity and message source authenticity in social media. To overcome vulnerabilities in ANITW, a part of the message hash and IBS-based signature is introduced as the watermark. The message integrity is verifiable by using part of the message hash as a watermark to detect any malicious. Moreover, implementing an IBS-based signature enables any reader to verify the message source’s authenticity easily. In addition, to improve ANITW performance in embedding capacity and invisibility, a novel bigram-based character encoding and embedding rule are also introduced. Based on the experiment and analysis result, it is proven that the proposed method can perform better than ANiTW in the security, embedding capacity, and invisibility aspects.

A Parallel Framework for Streaming Dimensionality Reduction.

The visualization of streaming high-dimensional data often needs to consider the speed in dimensionality reduction algorithms, the quality of visualized data patterns, and the stability of view graphs that usually change over time with new data. Existing methods of streaming high-dimensional data visualization primarily line up essential modules in a serial manner and often face challenges in satisfying all these design considerations. In this research, we propose a novel parallel framework for streaming high-dimensional data visualization to achieve high data processing speed, high quality in data patterns, and good stability in visual presentations. This framework arranges all essential modules in parallel to mitigate the delays caused by module waiting in serial setups. In addition, to facilitate the parallel pipeline, we redesign these modules with a parametric non-linear embedding method for new data embedding, an incremental learning method for online embedding function updating, and a hybrid strategy for optimized embedding updating. We also improve the coordination mechanism among these modules. Our experiments show that our method has advantages in embedding speed, quality, and stability over other existing methods to visualize streaming high-dimensional data.

A Knowledge Graph Embedding Framework With Triple Semantics

The knowledge graph embedding model aims to use low-dimensional real-valued vectors to represent the entities and relations in the triples, where operations such as link prediction and triple classification can be performed based on these representations. However, existing embedding models only consider the structural embedding of triples, while ignoring the semantic information of triples. This paper proposes a knowledge graph embedding learning framework combined with triple semantic information (KGSE). KGSE comprehensively considers the structural embedding and semantic embedding of triples, where semantic embedding is used as a supplement to improve the quality of embedding. Specifically, KGSE uses the improved TransD model to obtain the structural embedding of triples, and employs the deep convolutional neural model combined with an attention mechanism to obtain the semantic embedding of triples. In addition, a novel energy function is designed to jointly train the above two embeddings. Experimental results show that the proposed framework improves significantly compared with Trans-based models and other baseline models in link prediction and triple classification tasks, which verifies the effectiveness of the proposed framework.

Incorporating dynamic attention gating mechanism and pre-trained embedding for Chinese clinical named entity recognition

Clinical Named Entity Recognition (CNER) serves as a cornerstone in clinical research and healthcare informatics, with its primary goal being the discernment and categorization of clinical terminologies present within Electronic Medical Records (EMRs). While deep learning has significantly improved CNER's performance, these models still face challenges in identifying rare or unseen entities. Our study unveils an innovative DAGM-BiLSTM-CRF model that seamlessly integrates data-driven deep learning strategies with human knowledge to surmount this limitation. The distinctive features of this model lie in its integration of a pre-trained character embedding layer, adept at encapsulating nuanced semantic variations at the granularity of individual characters. Furthermore, it incorporates a bidirectional Long Short-Term Memory (BiLSTM) layer, proficient in deriving contextually relevant information by traversing the sequence from both endpoints. The most noteworthy aspect of our model is the inclusion of a Dynamic Attention Gating Mechanism (DAGM) that dynamically recalibrates attention weights based on the traits of the input sequence and a Conditional Random Fields (CRF) layer that considers interdependencies among output labels within a sequence. The evaluation was carried out on the CCKS-2017 CNER benchmark dataset to underscore the superiority of our model over existing cutting-edge methods, thereby validating its proficiency in deciphering the intricacies of Chinese electronic medical records.

Critical Relation Path Aggregation-Based Industrial Control Component Exploitable Vulnerability Reasoning

With the growing discovery of exposed vulnerabilities in the Industrial Control Components (ICCs), identification of the exploitable ones is urgent for Industrial Control System (ICS) administrators to proactively forecast potential threats. However, it is not a trivial task due to the complexity of the multi-source heterogeneous data and the lack of automatic analysis methods. To address these challenges, we propose an exploitability reasoning method based on the ICC-Vulnerability Knowledge Graph (KG) in which relation paths contain abundant potential evidence to support the reasoning. The reasoning task in this work refers to determining whether a specific relation is valid between an attacker entity and a possible exploitable vulnerability entity with the help of a collective of the critical paths. The proposed method consists of three primary building blocks: KG construction, relation path representation, and query relation reasoning. A security-oriented ontology combines exploit modeling, which provides a guideline for the integration of the scattered knowledge while constructing the KG. We emphasize the role of the aggregation of the attention mechanism in representation learning and ultimate reasoning. In order to acquire a high-quality representation, the entity and relation embeddings take advantage of their local structure and related semantics. Some critical paths are assigned corresponding attentive weights and then they are aggregated for the determination of the query relation validity. In particular, similarity calculation is introduced into a critical path selection algorithm, which improves search and reasoning performance. Meanwhile, the proposed algorithm avoids redundant paths between the given pairs of entities. Experimental results show that the proposed method outperforms the state-of-the-art ones in the aspects of embedding quality and query relation reasoning accuracy.

Strain-minimizing hyperbolic network embeddings with landmarks

Abstract We introduce L-hydra (landmarked hyperbolic distance recovery and approximation), a method for embedding network- or distance-based data into hyperbolic space, which requires only the distance measurements to a few ‘landmark nodes’. This landmark heuristic makes L-hydra applicable to large-scale graphs and improves upon previously introduced methods. As a mathematical justification, we show that a point configuration in $d$-dimensional hyperbolic space can be perfectly recovered (up to isometry) from distance measurements to just $d+1$ landmarks. We also show that L-hydra solves a two-stage strain-minimization problem, similar to our previous (unlandmarked) method ‘hydra’. Testing on real network data, we show that L-hydra is an order of magnitude faster than the existing hyperbolic embedding methods and scales linearly in the number of nodes. While the embedding error of L-hydra is higher than the error of the existing methods, we introduce an extension, L-hydra+, which outperforms the existing methods in both runtime and embedding quality.