Contextual Embeddings Research Articles

A Context-Enhanced De-identification System.

Many modern entity recognition systems, including the current state-of-the-art de-identification systems, are based on bidirectional long short-term memory (biLSTM) units augmented by a conditional random field (CRF) sequence optimizer. These systems process the input sentence by sentence. This approach prevents the systems from capturing dependencies over sentence boundaries and makes accurate sentence boundary detection a prerequisite. Since sentence boundary detection can be problematic especially in clinical reports, where dependencies and co-references across sentence boundaries are abundant, these systems have clear limitations. In this study, we built a new system on the framework of one of the current state-of-the-art de-identification systems, NeuroNER, to overcome these limitations. This new system incorporates context embeddings through forward and backward n -grams without using sentence boundaries. Our context-enhanced de-identification (CEDI) system captures dependencies over sentence boundaries and bypasses the sentence boundary detection problem altogether. We enhanced this system with deep affix features and an attention mechanism to capture the pertinent parts of the input. The CEDI system outperforms NeuroNER on the 2006 i2b2 de-identification challenge dataset, the 2014 i2b2 shared task de-identification dataset, and the 2016 CEGS N-GRID de-identification dataset (p < 0.01). All datasets comprise narrative clinical reports in English but contain different note types varying from discharge summaries to psychiatric notes. Enhancing CEDI with deep affix features and the attention mechanism further increased performance.

Stock movement prediction via gated recurrent unit network based on reinforcement learning with incorporated attention mechanisms

The recent advances usually mine market information from the chaotic data to conduct a stock movement prediction task. However, the current stock price movement prediction approaches mainly compute attention weighted sum of the global contextual semantic embeddings, which fails to combine local word-level or char-level ones to jointly learn news-level representation. Moreover, for Chinese stock price movement prediction task, some collected news texts are chaotic even irrelevant to the target stock. It suggests that the models need filter some news-level representations (viewed as noises) to enhance the performance. To that aim, we develop a novel stock price movement prediction network via bidirectional gated recurrent unit (GRU) network based on reinforcement learning (RL) with incorporated attention mechanism. In specific, to reduce the noise of news texts and learn news-level representation with more abundant semantics, two novel attention mechanisms respectively based on add and dot operation were first proposed in this work. We then design a novel GRU structure based on RL to filter some irrelated news-level representations (i.e., news-level noises) and capture abundant long-term dependencies. Finally, the experimental results show that the proposed model far outperforms the recent advances and achieves state-of-the-art performances.

ACE-ADP: Adversarial Contextual Embeddings Based Named Entity Recognition for Agricultural Diseases and Pests

Entity recognition tasks, which aim to utilize the deep learning-based models to identify the agricultural diseases and pests-related nouns such as the names of diseases, pests, and drugs from the texts collected on the internet or input by users, are a fundamental component for agricultural knowledge graph construction and question-answering, which will be implemented as a web application and provide the general public with solutions for agricultural diseases and pest control. Nonetheless, there are still challenges: (1) the polysemous problem needs to be further solved, (2) the quality of the text representation needs to be further enhanced, (3) the performance for rare entities needs to be further improved. We proposed an adversarial contextual embeddings-based model named ACE-ADP for named entity recognition in Chinese agricultural diseases and pests domain (CNER-ADP). First, we enhanced the text representation and overcame the polysemy problem by using the fine-tuned BERT model to generate the contextual character-level embedded representation with the specific knowledge. Second, adversarial training was also introduced to enhance the generalization and robustness in terms of identifying the rare entities. The experimental results showed that our model achieved an F1 of 98.31% with 4.23% relative improvement compared to the baseline model (i.e., word2vec-based BiLSTM-CRF) on the self-annotated corpus named Chinese named entity recognition dataset for agricultural diseases and pests (AgCNER). Besides, the ablation study and discussion demonstrated that ACE-ADP could not only effectively extract rare entities but also maintain a powerful ability to predict new entities in new datasets with high accuracy. It could be used as a basis for further research on other domain-specific named entity recognition.

Beyond visual semantics: Exploring the role of scene text in image understanding

Images with visual and scene text content are ubiquitous in everyday life. However, current image interpretation systems are mostly limited to using only the visual features, neglecting to leverage the scene text content. In this paper, we propose to jointly use scene text and visual channels for robust semantic interpretation of images. We do not only extract and encode visual and scene text cues, but also model their interplay to generate a contextual joint embedding with richer semantics. The contextual embedding thus generated is applied to retrieval and classification tasks on multimedia images, with scene text content, to demonstrate its effectiveness. In the retrieval framework, we augment our learned text-visual semantic representation with scene text cues, to mitigate vocabulary misses that may have occurred during the semantic embedding. To deal with irrelevant or erroneous recognition of scene text, we also apply query-based attention to our text channel. We show how the multi-channel approach, involving visual semantics and scene text, improves upon state of the art.

Recognizing Emotion Cause in Conversations

We address the problem of recognizing emotion cause in conversations, define two novel sub-tasks of this problem, and provide a corresponding dialogue-level dataset, along with strong transformer-based baselines. The dataset is available at https://github.com/declare-lab/RECCON . Recognizing the cause behind emotions in text is a fundamental yet under-explored area of research in NLP. Advances in this area hold the potential to improve interpretability and performance in affect-based models. Identifying emotion causes at the utterance level in conversations is particularly challenging due to the intermingling dynamics among the interlocutors. We introduce the task of Recognizing Emotion Cause in CONversations with an accompanying dataset named RECCON, containing over 1,000 dialogues and 10,000 utterance cause/effect pairs. Furthermore, we define different cause types based on the source of the causes, and establish strong Transformer-based baselines to address two different sub-tasks on this dataset. Our transformer-based baselines, which leverage contextual pre-trained embeddings, such as RoBERTa, outperform the state-of-the-art emotion cause extraction approaches on our dataset. We introduce a new task highly relevant for (explainable) emotion-aware artificial intelligence: recognizing emotion cause in conversations, provide a new highly challenging publicly available dialogue-level dataset for this task, and give strong baseline results on this dataset.

Leveraging Pre-Trained Contextualized Word Embeddings to Enhance Sentiment Classification of Drug Reviews

Traditionally, pharmacovigilance data are collected during clinical trials on a small sample of patients and are therefore insufficient to adequately assess drugs. Nowadays, consumers use online drug forums to share their opinions and experiences about medication. These feedbacks, which are widely available on the web, are automatically analyzed to extract relevant information for decision-making. Currently, sentiment analysis methods are being put forward to leverage consumers' opinions and produce useful drug monitoring indicators. However, these methods' effectiveness depends on the quality of word representation, which presents a real challenge because the information contained in user reviews is noisy and very subjective. Over time, several sentiment classification problems use machine learning methods based on the traditional bag of words model, sometimes enhanced with lexical resources. In recent years, word embedding models have significantly improved classification performance due to their ability to capture words' syntactic and semantic properties. Unfortunately, these latter models are weak in sentiment classification tasks because they are unable to encode sentiment information in the word representation. Indeed, two words with opposite polarities can have close word embeddings as they appear together in the same context. To overcome this drawback, some studies have proposed refining pre-trained word embeddings with lexical resources or learning word embeddings using training data. However, these models depend on external resources and are complex to implement. This work proposes a deep contextual word embeddings model called ELMo that inherently captures the sentiment information by providing separate vectors for words with opposite polarities. Different variants of our proposed model are compared with a benchmark of pre-trained word embeddings models using SVM classifier trained on Drug Review Dataset. Experimental results show that ELMo embeddings improve classification performance in sentiment analysis tasks on the pharmaceutical domain.

Unified concept and assertion detection using contextual multi-task learning in a clinical decision support system

Assertions, such as negation and speculation, alter the meaning of clinical findings (‘concepts’) in Electronic Health Records. Accurate assertion detection is vital to the identification of target findings in clinical decision support systems. Diverse clinical concepts and assertion modifiers embedded within longer sentences add to the challenge of error-free detection.Recent approaches leveraging biomedical contextual embeddings lead to standalone concept and assertion models that do not effectively utilize inter-task knowledge transfer. We propose a novel neural model integrating task-specific fine-tuning and multi-task learning in a coherent framework based on the hierarchical relationship between the tasks. We show that such a unified framework enhances both the tasks using several real-world clinical notes’ datasets (n2c2 2010, n2c2 2012, NegEx). Concept task performance enhanced by +1.69 F1 on n2c2 2010 and +2.96 F1 on n2c2 2012 compared to standalone baselines. Assertion recognition improved by +2.89 F1 and +3.77 F1, respectively. Negation detection under low-resource settings increased significantly (+2.4 F1, p-value = 3.11E−05, McNemar’s test), demonstrating the impact of inter-task knowledge transfer. The integrated architecture enhanced the generalization performance of speculation detection (+2.09 F1).To the best of our knowledge, this model is the first demonstration of a contextual multi-task system for unified detection of concepts and assertions in clinical decision support applications.

The Integration of Linguistic and Geospatial Features Using Global Context Embedding for Automated Text Geocoding

Geocoding is an essential procedure in geographical information retrieval to associate place names with coordinates. Due to the inherent ambiguity of place names in natural language and the scarcity of place names in textual data, it is widely recognized that geocoding is challenging. Recent advances in deep learning have promoted the use of the neural network to improve the performance of geocoding. However, most of the existing approaches consider only the local context, e.g., neighboring words in a sentence, as opposed to the global context, e.g., the topic of the document. Lack of global information may have a severe impact on the robustness of the model. To fill the research gap, this paper proposes a novel global context embedding approach to generate linguistic and geospatial features through topic embedding and location embedding, respectively. A deep neural network called LGGeoCoder, which integrates local and global features, is developed to solve the geocoding as a classification problem. The experiments on a Wikipedia place name dataset demonstrate that LGGeoCoder achieves competitive performance compared with state-of-the-art models. Furthermore, the effect of introducing global linguistic and geospatial features in geocoding to alleviate the ambiguity and scarcity problem is discussed.

Diversified text-to-image generation via deep mutual information estimation

Generating photo-realistic, text-matched, and diverse images simultaneously from given text descriptions is a challenging task in computer vision. Previous works mostly focus on visual realism and semantic relevance, but neglect variations of the generated results which is also an important target in text-to-image generation (T2I). In this paper, we design a new module that improves the training of generative adversarial nets (GANs) to generate diverse images from a text input. Our T2I method is based on conditional GANs and has three components: contextual text embedding module (CTEM), deep Mutual Information (MI) estimation module for stacked image generation (DMIEM), and text-image semantic relevance module (TISRM). CTEM attempts to learn text embeddings via leveraging the fine-tuning capabilities of BERT. DMIEM has a stack of attention embedded generators, integrating with global/local MI estimation and maximization between input and output of the generator to make the T2I mapping more relevant, and then generating diverse and photo-realistic images progressively. TISRM is introduced to enhance the semantic consistency of the text-image pairs by regenerating the text descriptions from the generated images. Extensive experiments on three datasets indicate that our method can generate text-matched and more diverse images without quality degradation compared to the state-of-the-art approaches.

Constructivist Approaches to First Language Acquisition

Constructivist approaches to language acquisition predict that form-function mappings are derived from distributional patterns in the input, and their contextual embedding. This requires a detailed analysis of the input, and the integration of information from different contingencies. Regarding the acquisition of morphology, it is shown which types of information leads to the induction of (lexical) categories, and to paradigm building. Regarding the acquisition of word order, it is shown how languages with fixed or variable word order profit from stable syntactic hyperschemas, but require a more detailed analyses of the form-function contingencies to identify the underlying, more specific semantic, syntactic and morphological patterns. At a theoretical level, it is shown how findings from acquisition and processing converge into new linguistic theories that aim to account for regular as well as irregular phenomena in language.

Dual-Channel Reasoning Model for Complex Question Answering

Multihop question answering has attracted extensive studies in recent years because of the emergence of human annotated datasets and associated leaderboards. Recent studies have revealed that question answering systems learn to exploit annotation artifacts and other biases in current datasets. Therefore, a model with strong interpretability should not only predict the final answer, but more importantly find the supporting facts’ sentences necessary to answer complex questions, also known as evidence sentences. Most existing methods predict the final answer and evidence sentences in sequence or simultaneously, which inhibits the ability of models to predict the path of reasoning. In this paper, we propose a dual-channel reasoning architecture, where two reasoning channels predict the final answer and supporting facts’ sentences, respectively, while sharing the contextual embedding layer. The two reasoning channels can simply use the same reasoning structure without additional network designs. Through experimental analysis based on public question answering datasets, we demonstrate the effectiveness of our proposed method

Innovation logics in the digital era: a systemic review of the emerging digital innovation regime

ABSTRACT This essay reviews the ontological status of digital material in industrial operations and conditions it provides for innovation. It recognises the semiotic quality of digital material and the critical role of von Neumann architecture as to show that digital innovation advances through a three pronged, largely orthogonal process of embedding- a process of interlacing elements of one innovation domain to that of another. Three types of embedding define digital innovation: operational embedding (code-computer), virtual embedding (real-world phenomena-code) and contextual embedding (performing code-social setting). Each constitutes a ‘leverage point’ for further expanding digital innovation and they interact dynamically while industrial innovation is defined by a static virtual and contextual embedding. The processes that underlie innovation in the two regimes follow differential logics. In industrial regime (1) discover-synthesise and (2) manufacture-distribute activities combine to form the leverage points for innovation. In digital regime 1) discover product or behaviour (ideation); 2) abstract to digital material (virtualise); 3) Implement and replicate digital material (variations in operational/contextual embedding); and 4) Deliver product and/or behaviour (perform digital material) activities form the leverage points for innovation enabling wider, open, and faster innovation with a distinct value logic.

Arabic Biomedical Community Question Answering Based on Contextualized Embeddings

Community question answering has become increasingly important as they are practical for seeking and sharing information. Applying deep learning models often leads to good performance, but it requires an extensive amount of annotated data, a problem exacerbated for languages suffering a scarcity of resources. Contextualized language representation models have gained success due to promising results obtained on a wide array of downstream natural language processing tasks such as text classification, textual entailment, and paraphrase identification. This paper presents a novel approach by fine-tuning contextualized embeddings for a medical domain community question answering task. The authors propose an architecture combining two neural models powered by pre-trained contextual embeddings to learn a sentence representation and thereafter fine-tuned on the task to compute a score used for both ranking and classification. The experimental results on SemEval Task 3 CQA show that the model significantly outperforms the state-of-the-art models by almost 2% for the '16 edition and 1% for the '17 edition.

BERT syntactic transfer: A computational experiment on Italian, French and English languages

This paper investigates the ability of multilingual BERT (mBERT) language model to transfer syntactic knowledge cross-lingually, verifying if and to which extent syntactic dependency relationships learnt in a language are maintained in other languages. In detail, the main contributions of this paper are: (i) an analysis of the cross-lingual syntactic transfer capability of mBERT model; (ii) a detailed comparison of cross-language syntactic transfer among languages belonging to different branches of the Indo-European languages, namely English, Italian and French, which present very different syntactic constructions; (iii) a study on the transferability of a syntactic phenomenon peculiar of Italian language, namely the pronoun dropping (pro-drop), also known as omissibility of the subject. To this end, a structural probe devoted to reconstruct the dependency parse tree of a sentence has been exploited, representing the input sentences with the contextual embeddings from mBERT layers. The results of the experimental assessment have shown a transfer of syntactic knowledge of the mBERT model among these languages. Moreover, the behaviour of the probe in the transition from pro-drop to non-pro-drop languages and vice versa has proven to be more effective in case of languages sharing a common linguistic matrix. The possibility of transferring syntactical knowledge, especially in the case of specific phenomena, meets both a theoretical need and can have important practical implications in syntactic tasks, such as dependency parsing.

Sentiment Analysis Using XLM-R Transformer and Zero-shot Transfer Learning on Resource-poor Indian Language

Sentiment analysis on social media relies on comprehending the natural language and using a robust machine learning technique that learns multiple layers of representations or features of the data and produces state-of-the-art prediction results. The cultural miscellanies, geographically limited trending topic hash-tags, access to aboriginal language keyboards, and conversational comfort in native language compound the linguistic challenges of sentiment analysis. This research evaluates the performance of cross-lingual contextual word embeddings and zero-shot transfer learning in projecting predictions from resource-rich English to resource-poor Hindi language. The cross-lingual XLM-RoBERTa classification model is trained and fine-tuned using the English language Benchmark SemEval 2017 dataset Task 4 A and subsequently zero-shot transfer learning is used to evaluate the classification model on two Hindi sentence-level sentiment analysis datasets, namely, IITP-Movie and IITP-Product review datasets. The proposed model compares favorably to state-of-the-art approaches and gives an effective solution to sentence-level (tweet-level) analysis of sentiments in a resource-poor scenario. The proposed model compares favorably to state-of-the-art approaches and achieves an average performance accuracy of 60.93 on both the Hindi datasets.

A Sentiment-Aware Contextual Model for Real-Time Disaster Prediction Using Twitter Data

The massive amount of data generated by social media present a unique opportunity for disaster analysis. As a leading social platform, Twitter generates over 500 million Tweets each day. Due to its real-time characteristic, more agencies employ Twitter to track disaster events to make a speedy rescue plan. However, it is challenging to build an accurate predictive model to identify disaster Tweets, which may lack sufficient context due to the length limit. In addition, disaster Tweets and regular ones can be hard to distinguish because of word ambiguity. In this paper, we propose a sentiment-aware contextual model named SentiBERT-BiLSTM-CNN for disaster detection using Tweets. The proposed learning pipeline consists of SentiBERT that can generate sentimental contextual embeddings from a Tweet, a Bidirectional long short-term memory (BiLSTM) layer with attention, and a 1D convolutional layer for local feature extraction. We conduct extensive experiments to validate certain design choices of the model and compare our model with its peers. Results show that the proposed SentiBERT-BiLSTM-CNN demonstrates superior performance in the F1 score, making it a competitive model in Tweets-based disaster prediction.

Age-related changes in time perception: The impact of naturalistic environments and retrospective judgements on timing performance.

Reduced timing abilities have been reported in older adults and are associated with pathological cognitive decline. However, time perception experiments often lack ecological validity. Especially the reduced complexity of experimental stimuli and the participants’ awareness of the time-related nature of the task can influence lab-assessed timing performance and thereby conceal age-related differences. An approximation of more naturalistic paradigms can provide important information about age-related changes in timing abilities. To determine the impact of higher ecological validity on timing experiments, we implemented a paradigm that allowed us to test (1) the effect of embedding the to-be-timed stimuli within a naturalistic visual scene and (2) the effect of retrospective time judgements, which are more common in real life than prospective judgements. The results show that compared with out-of-context stimuli, younger adults benefit from a naturalistic embedding of stimuli (reflected in higher precision and less errors), whereas the performance of older adults is reduced when confronted with naturalistic stimuli. Differences between retrospective and prospective time judgements were not modulated by age. We conclude that, potentially driven by difficulties in suppressing temporally irrelevant environmental information, the contextual embedding of naturalistic stimuli can affect the degree to which age influences the performance in time perception tasks.

Visualizing Carotid Blood Flow Simulations for Stroke Prevention

AbstractIn this work, we investigate how concepts from medical flow visualization can be applied to enhance stroke prevention diagnostics. Our focus lies on carotid stenoses, i.e., local narrowings of the major brain‐supplying arteries, which are a frequent cause of stroke. Carotid surgery can reduce the stroke risk associated with stenoses, however, the procedure entails risks itself. Therefore, a thorough assessment of each case is necessary. In routine diagnostics, the morphology and hemodynamics of an afflicted vessel are separately analyzed using angiography and sonography, respectively. Blood flow simulations based on computational fluid dynamics could enable the visual integration of hemodynamic and morphological information and provide a higher resolution on relevant parameters. We identify and abstract the tasks involved in the assessment of stenoses and investigate how clinicians could derive relevant insights from carotid blood flow simulations. We adapt and refine a combination of techniques to facilitate this purpose, integrating spatiotemporal navigation, dimensional reduction, and contextual embedding. We evaluated and discussed our approach with an interdisciplinary group of medical practitioners, fluid simulation and flow visualization researchers. Our initial findings indicate that visualization techniques could promote usage of carotid blood flow simulations in practice.

A contrastive study on thanking in Arabic dialects and English

Cross-Cultural Pragmatics (CCP) is a field of study that focuses on the interaction of people from various cultures. It is to clarify cultural distinctions between languages that acquire meaning through context and sociocultural embedding, resulting in a pragmatic communication failure. This study determines whether the Arabic language, because of dialectical variation, contains more thanking methods than English. Additionally, it aims to investigate the face-threatening strategies used by Arabs and native English speakers. Besides, it aims to determine whether contextual variables affect thanking expressions. The data collection instrument was an open-ended questionnaire in two versions Arabic and English. The results were then analyzed quantitatively using SPSS software version 26.0, following Cheng's classification of thanking. The findings indicate that dialectical diversity is not associated with an increase in thanking strategies that are more prevalent in English. Additionally, religion affects the Arabs speaking, their manner of thanking does not exclude religious expressions, which are their preferred method of expressing their politeness and gratitude. In terms of face-threatening strategies, both native Arabic and English speakers preferred negative politeness to positive politeness, which focuses on minimizing the FTA's particular imposition. Concerning contextual variables such as familiarity and social status, both affect how the thanking speech act is performed. It will be worthwhile to investigate thanking in Arabic and English with a specific factor such as social distance. Additionally, it is beneficial to examine thanking in Arabic dialects regarding civilization's cultural influence and the dialect's proximity to standard Arabic.

Long short-term memory network for learning sentences similarity using deep contextual embeddings

Long short-term memory network for learning sentences similarity using deep contextual embeddings