Embedding Quality Research Articles

Unsupervised Detection of Distinctive Regions on 3D Shapes

This article presents a novel approach to learn and detect distinctive regions on 3D shapes. Unlike previous works, which require labeled data, our method is unsupervised. We conduct the analysis on point sets sampled from 3D shapes, then formulate and train a deep neural network for an unsupervised shape clustering task to learn local and global features for distinguishing shapes with respect to a given shape set. To drive the network to learn in an unsupervised manner, we design a clustering-based nonparametric softmax classifier with an iterative re-clustering of shapes, and an adapted contrastive loss for enhancing the feature embedding quality and stabilizing the learning process. By then, we encourage the network to learn the point distinctiveness on the input shapes. We extensively evaluate various aspects of our approach and present its applications for distinctiveness-guided shape retrieval, sampling, and view selection in 3D scenes.

Ten minutes with James Mountford, Chief of Quality and Learning, NHS Nightingale Hospital London

![Graphic][1] ### Biography James Mountford is Director of Quality at the Royal Free London NHS Foundation Trust, which has a multiyear programme focused on embedding quality and value improvement into...

A simple Galois Power-of-Two real time embedding scheme for performing Arabic morphology deep learning tasks

This paper describes how a simple novel Galois Power-of-Two (GPOW2) real-time embedding scheme is used to improve the performance and accuracy of downstream NLP tasks. GPOW2 computes embeddings live on the fly (real time) in the context of target NLP tasks without the need for tabulated pre-embeddings. One excellent feature of the method is the ability to capture multilevel embeddings in the same pass. It simultaneously computes character, word and sentence embeddings on the fly. GPOW2 has been derived in the context of attempts to improve the performance of the SWAM Arabic morphological engine, which is a multipurpose tool that supports segmentation, classification, POS tagging, spell checking, word embeddings, sematic search, among other tasks. SWAM is a pattern-oriented algorithm that relies on morphological patterns and POS tagging to perform NLP tasks. The paper demonstrates how GPOW2 led to improvements in the accuracy of POS tagging and pattern matching, and accordingly the performance of the whole engine. The accuracy for pattern prediction is 99.47% and is 98.80% for POS tagging.

High-Capacity Data Hiding for ABTC-EQ Based Compressed Image

We present a new data hiding method based on Adaptive BTC Edge Quantization (ABTC-EQ) using an optimal pixel adjustment process (OPAP) to optimize two quantization levels. The reason we choose ABTC-EQ as a cover media is that it is superior to AMBTC in maintaining a high-quality image after encoding is executed. ABTC-EQ is represented by a form of t r i o ( Q 1 , Q 2 , [ Q 3 ] , BM) where Q is quantization levels ( Q 1 ≤ Q 2 ≤ Q 3 ) , and BM is a bitmap). The number of quantization levels are two or three, depending on whether the cover image has an edge or not. Before embedding secret bits in every block, we categorize every block into smooth block or complex block by a threshold. In case a block size is 4x4, the sixteen secret bits are replaced by a bitmap of the smooth block for embedding a message directly. On the other hand, OPAP method conceals 1 bit into LSB and 2LSB respectively, and maintains the quality of an image as a way of minimizing the errors which occur in the embedding procedure. The sufficient experimental results demonsrate that the performance of our proposed scheme is satisfactory in terms of the embedding capacity and quality of an image.

Neural negated entity recognition in Spanish electronic health records.

This work deals with negation detection in the context of clinical texts. Negation detection is a key for decision support systems since negated events (detection of absence of some events) help ascertain current medical conditions. For artificial intelligence, negation detection is a valuable point as it can revert the meaning of a part of a text and, accordingly, influence other tasks such as medical dosage adjustment, the detection of adverse drug reactions or hospital acquired diseases. We focus on negated medical events such as disorders, findings and allergies. From Natural Language Processing (NLP) background, we refer to them as negated medical entities. A novelty of this work is that we approached this task as Named Entity Recognition (NER) with the restriction that just negated medical entities must be recognized (in an attempt to help distinguish them from non-negated ones). Our study is driven with Electronic Health Records (EHRs) written in Spanish. A challenge to cope with is the lexical variability (alternative medical forms, abbreviations, etc.). To this end, we employed an approach based on deep learning. Specifically, the system combines character embeddings to cope with out-of-vocabulary (OOV) words, Long Short-Term Memory (LSTM) networks to model contextual representations and it makes use of Conditional Random Fields (CRF) to classify each medical entity as either negated or not given the contextual dense representation. Moreover, we explored both embeddings created from words and embeddings created from lemmas. The best results were obtained with the lemmatized embeddings. Apparently, this approach reinforced the capability of the LSTMs to cope with the high lexical variability. The f-measure for exact-match was 65.1 and 82.4 for the partial-match.

Joint Character-Level Word Embedding and Adversarial Stability Training to Defend Adversarial Text

Text classification is a basic task in natural language processing, but the small character perturbations in words can greatly decrease the effectiveness of text classification models, which is called character-level adversarial example attack. There are two main challenges in character-level adversarial examples defense, which are out-of-vocabulary words in word embedding model and the distribution difference between training and inference. Both of these two challenges make the character-level adversarial examples difficult to defend. In this paper, we propose a framework which jointly uses the character embedding and the adversarial stability training to overcome these two challenges. Our experimental results on five text classification data sets show that the models based on our framework can effectively defend character-level adversarial examples, and our models can defend 93.19% gradient-based adversarial examples and 94.83% natural adversarial examples, which outperforms the state-of-the-art defense models.

Learning Signed Network Embedding via Graph Attention

Learning the low-dimensional representations of graphs (i.e., network embedding) plays a critical role in network analysis and facilitates many downstream tasks. Recently graph convolutional networks (GCNs) have revolutionized the field of network embedding, and led to state-of-the-art performance in network analysis tasks such as link prediction and node classification. Nevertheless, most of the existing GCN-based network embedding methods are proposed for unsigned networks. However, in the real world, some of the networks are signed, where the links are annotated with different polarities, e.g., positive vs. negative. Since negative links may have different properties from the positive ones and can also significantly affect the quality of network embedding. Thus in this paper, we propose a novel network embedding framework SNEA to learn Signed Network Embedding via graph Attention. In particular, we propose a masked self-attentional layer, which leverages self-attention mechanism to estimate the importance coefficient for pair of nodes connected by different type of links during the embedding aggregation process. Then SNEA utilizes the masked self-attentional layers to aggregate more important information from neighboring nodes to generate the node embeddings based on balance theory. Experimental results demonstrate the effectiveness of the proposed framework through signed link prediction task on several real-world signed network datasets.

Semantics-Aware BERT for Language Understanding

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

Robust Named Entity Recognition with Truecasing Pretraining

Although modern named entity recognition (NER) systems show impressive performance on standard datasets, they perform poorly when presented with noisy data. In particular, capitalization is a strong signal for entities in many languages, and even state of the art models overfit to this feature, with drastically lower performance on uncapitalized text. In this work, we address the problem of robustness of NER systems in data with noisy or uncertain casing, using a pretraining objective that predicts casing in text, or a truecaser, leveraging unlabeled data. The pretrained truecaser is combined with a standard BiLSTM-CRF model for NER by appending output distributions to character embeddings. In experiments over several datasets of varying domain and casing quality, we show that our new model improves performance in uncased text, even adding value to uncased BERT embeddings. Our method achieves a new state of the art on the WNUT17 shared task dataset.

Exploiting Bi-LSTMs for Named Entity Recognition in Indian Culinary Science

This paper discusses the use of Bidirectional LSTMs for recognition of Named Entities over the Indian Recipe Blogs. Recipe posts from popular blogs including Hebbar's Kitchen are harvested and trained for recognizing NEs. Both the word embeddings and character embeddings are utilized as feature vectors for training the Bi-LSTM. CRF model is used for joint decoding of the labels. The system shows a development data F1 score of 92.87% and test data F1 score of 94.66%. The dataset used and meta-results obtained are released freely for research use.

Research on Chinese medical named entity recognition based on collaborative cooperation of multiple neural network models.

Medical named entity recognition (NER) in Chinese electronic medical records (CEMRs) has drawn much research attention, and plays a vital prerequisite role for extracting high-value medical information. In 2018, China Health Information Processing Conference (CHIP2018) organized a medical NER academic competition aiming to extract three types of malignant tumor entity from CEMRs. Since the three types of entity are highly domain-specific and interdependency, extraction of them cannot be achieved with a single neural network model. Based on comprehensive study of the three types of entity and the entity interdependencies, we propose a collaborative cooperation of multiple neural network models based approach, which consists of two BiLSTM-CRF models and a CNN model. In order to tackle the problem that target scene dataset is small and entity distributions are sparse, we introduce non-target scene datasets and propose sentence-level neural network model transfer learning. Based on 30,000 real-world CEMRs, we pre-train medical domain-specific Chinese character embeddings with word2vec, GloVe and ELMo, and apply them to our approach respectively to validate effects of pre-trained language models in Chinese medical NER. Also, as control experiments, we apply Gated Recurrent Unit to our approach. Finally, our approach achieves an overall F1-score of 87.60%, which is the state-of-the-art performance to the best of our knowledge. In addition, our approach has won the champion of the medical NER academic competition organized by 2019 China Conference on Knowledge Graph and Semantic Computing, which proves the outstanding generalization ability of our approach.

Hydra: a method for strain-minimizing hyperbolic embedding of network- and distance-based data

Abstract We introduce hydra (hyperbolic distance recovery and approximation), a new method for embedding network- or distance-based data into hyperbolic space. We show mathematically that hydra satisfies a certain optimality guarantee: it minimizes the ‘hyperbolic strain’ between original and embedded data points. Moreover, it is able to recover points exactly, when they are contained in a low-dimensional hyperbolic subspace of the feature space. Testing on real network data we show that the embedding quality of hydra is competitive with existing hyperbolic embedding methods, but achieved at substantially shorter computation time. An extended method, termed hydra+, typically outperforms existing methods in both computation time and embedding quality.

Leveraging multiple features for document sentiment classification

Sentiment classification is an important research task in Natural Language Processing. To fulfill this type of classification, previous works have focused on leveraging task-specific features. However, they only notice part of the related features. Also, state-of-the-art methods based on neural networks often ignore traditional features. This paper proposes a novel text sentiment classification method that learns the representation of texts by hierarchically incorporating multiple features. More specifically, we design different representations for sentiment words according to the polarity of labeled texts and whether negation exists; we distinguish words with different part-of-speech tags; emoticons, if there are, are to optimize the word vectors obtained in the previous step; apart from word embeddings, character embeddings are also trained. We use a deep neural network to get a sentence-level representation from both word and character sequence. For documents with at least two sentences, we use a hierarchical structure and design a rule to give more weight to import sentences empirically to get a document-level representation. Experimental results on open datasets demonstrate that our method could effectively improve the sentiment classification performance compared with the basic models and state-of-the-art methods.

A Hybrid Classification Method via Character Embedding in Chinese Short Text With Few Words

Last decades have witnessed the significance development of research in short text classification. However, most existing methods only focus on the text which contained dozens of words like Twitter or MicroBlog, but not take the short text with few words like news headline or invoice name into consideration. Meanwhile, contemporary short text classification methods either to expand feature of short text with external corpus or to learn the feature representation from all the texts, which have not take the difference between words of short text into full consideration. Notably, the classification of short text with few words are usually determined by a few specific key words contrary to documents classification or traditional short text classification. To address these problems, this paper propose a hybrid classification method of Attention mechanism and Feature selection via Character embedding in Chinese short text with few words, called AFC. More specifically, firstly, the character embedding is computed to represent Chinese short texts with few words, which takes full advantage of short text information without external corpus. Secondly, attention-based LSTM is introduced in our method to project the data into feature representation space with weighting, which make the keywords in classification have more subtle value. Furthermore, the semantic similarity between content and class label information is calculated for feature selection, which reduces the possible negative influence of some redundant information on classification. Experiments on real-world datasets demonstrate the effectiveness of our method compared to other competing methods.

Improving Online Clustering of Chinese Technology Web News With Bag-of-Near-Synonyms

In the Internet era, online clustering of technology web news can help discover scientific breakthroughs and grasp technology trends. To do that automatically, the news documents to be clustered must be represented appropriately with numerical vectors. However, traditional representations such as Term Frequency-Inverse Document Frequency (TF-IDF) cannot distinguish near-synonyms and may cause “dimension disaster.” To overcome these problems, this article proposes the Bag-of-Near-Synonyms (BoNS) model based on the idea to construct near-synonym sets using word embeddings and agglomerative clustering, and then to represent a document with a Set Frequency-Inverse Document Frequency (SF-IDF) vector in which each dimension corresponds to a near-synonym set rather than a single word. To speed up computation, we further propose the hashed version of SF-IDF and name it hSF-IDF, which employs a hash function to map each near-synonym set to a unique number as the key and hence reduces the computation of SF to linear time. In addition, we apply hSF-IDF to online clustering of Chinese technology web news and propose an improved batch-based method. Extensive experiments have been conducted on a real-world dataset. The results show that our model outperforms some strong baselines including TF-IDF, average pooling of word or character embeddings, Latent Dirichlet Allocation (LDA), and bag-of-concepts in terms of both accuracy and efficiency.

Suicidal Ideation Cause Extraction From Social Texts

Suicide has become a major public health and social concern in the world. Suicidal ideation cause extraction (SICE) in social texts can provide support for suicide prevention. This article summarizes the research on suicidal ideation causes (SICs) through the use of psychological and sociological analysis. Then, a social text-based SIC dataset is constructed and analyzed statistically for various features. A CRF model is provided along with Char-BiLSTM-CRF, which uses concatenation of word embeddings and character embeddings as word representation inputs. Then, the effects on the task are explored by the word (W), part of speech (POS), dependence relationship (DP), suicidal psychology (PCS), emotion (ET), and language (LG) features in the CRF model. The experiment shows that the word features worked best. POS and DP can be somehow covered by word features. PCS, ET and LG features can improve the effect of SICE. It also shows that Char-BiLSTM-CRF is better than CRF in general, but CRF still has advantages in terms of precision. Adding character embeddings and CRF layers can significantly improve the extraction using Char-BiLSTM-CRF. The experiment also compared three-word embeddings with Word2vec, ELMo and BERT. Compared with Word2vec, the F-value of ELMo is increased by approximately 5% on average, and compared with ELMo, the C_F and E_F of BERT are increased by 3.5% and 2.3%, respectively. Finally, the challenge of SICE is discussed based on the experimental results of Char-BiLSTM-CRF.

Capsule Networks With Word-Attention Dynamic Routing for Cultural Relics Relation Extraction

Online museums and online cultural relic information provide abundant data for relation extraction research. However, in the relation extraction task of modelling space information, spatially insensitive methods of convolutional neural networks and long short term memory network in most current works still remain challenging in rich text structures, which makes models difficult to encode effectively and lacks the ability of text expression. To address this issue, we propose a framework named WAtt-Capsnet (the capsule network with word-attention dynamic routing), which is based on capsule networks with word-attention dynamic routing for the relation extraction task of online cultural relic data for capturing richer instantiation features. We further present combination embedding for capturing the characteristic information of Chinese sentences by considering the contribution of word embedding, parts of speech, character embedding and the position of words to capture rich internal structure information of sentences. More importantly, to reduce the decay of useful information in long sentences, we propose a routing algorithm based on a word-attention mechanism to focus on informative words. The experimental results demonstrate that the proposed method achieves significant performance for the relation extraction task of online cultural relic data.

Hierarchical Character Embeddings: Learning Phonological and Semantic Representations in Languages of Logographic Origin Using Recursive Neural Networks

Logographs (Chinese characters) have recursive structures (i.e. hierarchies of sub-units in logographs) that contain phonological and semantic information, as developmental psychology literature suggests that native speakers leverage on the structures to learn how to read. Exploiting these structures could potentially lead to better embeddings that can benefit many downstream tasks. We propose building hierarchical logograph (character) embeddings from logograph recursive structures using treeLSTM, a recursive neural network. Using recursive neural network imposes a prior on the mapping from logographs to embeddings since the network must read in the sub-units in logographs according to the order specified by the recursive structures. Based on human behavior in language learning and reading, we hypothesize that modeling logographs' structures using recursive neural network should be beneficial. To verify this claim, we consider two tasks (1) predicting logographs' Cantonese pronunciation from logographic structures and (2) language modeling. Empirical results show that the proposed hierarchical embeddings outperform baseline approaches. Diagnostic analysis suggests that hierarchical embeddings constructed using treeLSTM is less sensitive to distractors, thus is more robust, especially on complex logographs.

Embedding information reversibly in medical images for e-health

Embedding information in medical images is considered as one of the significant methods for safeguarding the integrity and authenticity of medical images besides providing security to electronic patient records (EPR). The conventional embedding methods deteriorate the perceptual quality of medical images making them unsuitable for proper diagnosis. To preserve the perceptual quality of medical images reversible embedding is used. The reversible embedding schemes, however, have less embedding capacity. In this work, a reversible scheme based on histogram bin shifting and RGB plane concatenation has been proposed which offers high embedding capacity as well. We have exploited the fact that medical images, unlike general images, consist of a large number of peaks and zero points that can be employed for reversibly embedding the data. Reversibility ensures that original image restoration takes place after the extraction of embedded data, which is of great importance in medical images for proper diagnosis and treatment. We have used various subjective and objective image quality metrics for analyzing the scheme. The proposed scheme has been shown to provide a Peak Signal to Noise Ratio (PSNR) value of above 56 dB for an embedding capacity of 0.58 bits per pixel (bpp). The results obtained show that the performance of scheme presented is far better in comparison to the state-of-the-art.

Enhancing Aspect-Based Sentiment Analysis With Capsule Network

Existing feature-based neural approaches for aspect-based sentiment analysis (ABSA) try to improve their performance with pre-trained word embeddings and by modeling the relations between the text sequence and the aspect (or category), thus heavily depending on the quality of word embeddings and task-specific architectures. Although the recently pre-trained language models, i.e., BERT and XLNet, have achieved state-of-the-art performance in a variety of natural language processing (NLP) tasks, they still subject to the aspect-specific, local feature-aware and task-agnostic challenges. To address these challenges, this paper proposes a XLNet and capsule network based model XLNetCN for ABSA. XLNetCN firstly constructs auxiliary sentence to model the sequence-aspect relation and generate global aspect-specific representations, which enables to enhance aspect-awareness and ensure the full pre-training of XLNet for improving task-agnostic capability. After that, XLNetCN also employs a capsule network with the dynamic routing algorithm to extract the local and spatial hierarchical relations of the text sequence, and yield its local feature representations, which are then merged with the global aspect-related representations for downstream classification via a softmax classifier. Experimental results show that XLNetCN outperforms significantly than the classical BERT, XLNet and traditional feature-based approaches on the two benchmark datasets of SemEval 2014, Laptop and Restaurant, and achieves new state-of-the-art results.