Embeddings From Language Models Research Articles

Multi-step Transfer Learning in Natural Language Processing for the Health Domain

The restricted access to data in healthcare facilities due to patient privacy and confidentiality policies has led to the application of general natural language processing (NLP) techniques advancing relatively slowly in the health domain. Additionally, because clinical data is unique to various institutions and laboratories, there are not enough standards and conventions for data annotation. In places without robust death registration systems, the cause of death (COD) is determined through a verbal autopsy (VA) report. A non-clinician field agent completes a VA report using a set of standardized questions as guide to identify the symptoms of a COD. The narrative text of the VA report is used as a case study to examine the difficulties of applying NLP techniques to the healthcare domain. This paper presents a framework that leverages knowledge across multiple domains via two domain adaptation techniques: feature extraction and fine-tuning. These techniques aim to improve VA text representations for COD classification tasks in the health domain. The framework is motivated by multi-step learning, where a final learning task is realized via a sequence of intermediate learning tasks. The framework builds upon the strengths of the Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Models (ELMo) models pretrained on the general English and biomedical domains. These models are employed to extract features from the VA narratives. Our results demonstrate improved performance when initializing the learning of BERT embeddings with ELMo embeddings. The benefit of incorporating character-level information for learning word embeddings in the English domain, coupled with word-level information for learning word embeddings in the biomedical domain, is also evident.

Enriching Urdu NER with BERT Embedding, Data Augmentation, and Hybrid Encoder-CNN Architecture

Named Entity Recognition (NER) is an indispensable component of Natural Language Processing (NLP), which aims to identify and classify entities within text data. While Deep Learning (DL) models have excelled in NER for well-resourced languages such as English, Spanish, and Chinese, they face significant hurdles when dealing with low-resource languages such as Urdu. These challenges stem from the intricate linguistic characteristics of Urdu, including morphological diversity, a context-dependent lexicon, and the scarcity of training data. This study addresses these issues by focusing on Urdu Named Entity Recognition (U-NER) and introducing three key contributions. First, various pre-trained embedding methods are employed, encompassing Word2vec (W2V), GloVe, FastText, Bidirectional Encoder Representations from Transformers (BERT), and Embeddings from language models (ELMo). In particular, fine-tuning is performed on BERT BASE and ELMo using Urdu Wikipedia and news articles. Second, a novel generative Data Augmentation (DA) technique replaces Named Entities (NEs) with mask tokens, employing pre-trained masked language models to predict masked tokens, effectively expanding the training dataset. Finally, the study introduces a novel hybrid model combining a Transformer Encoder with a Convolutional Neural Network (CNN) to capture the intricate morphology of Urdu. These modules enable the model to handle polysemy, extract short- and long-range dependencies, and enhance learning capacity. Empirical experiments demonstrate that the proposed model, incorporating BERT embeddings and an innovative DA approach, attains the highest F1-score of 93.99%, highlighting its efficacy for the U-NER task.

Evaluating sentiment analysis and word embedding techniques on Brexit

<p>In this study, we investigate the effectiveness of pre-trained word embeddings for sentiment analysis on a real-world topic, namely Brexit. We compare the performance of several popular word embedding models such global vectors for word representation (GloVe), FastText, word to vec (word2vec), and embeddings from language models (ELMo) on a dataset of tweets related to Brexit and evaluate their ability to classify the sentiment of the tweets as positive, negative, or neutral. We find that pre-trained word embeddings provide useful features for sentiment analysis and can significantly improve the performance of machine learning models. We also discuss the challenges and limitations of applying these models to complex, real-world texts such as those related to Brexit.</p><p> </p>

Development and Practical Applications of Computational Intelligence Technology

Computational intelligence (CI) uses applied computational methods for problem-solving inspired by the behavior of humans and animals. Biological systems are used to construct software to solve complex problems, and one type of such system is an artificial immune system (AIS), which imitates the immune system of a living body. AISs have been used to solve problems that require identification and learning, such as computer virus identification and removal, image identification, and function optimization problems. In the body’s immune system, a wide variety of cells work together to distinguish between the self and non-self and to eliminate the non-self. AISs enable learning and discrimination by imitating part or all of the mechanisms of a living body’s immune system. Certainly, some deep neural networks have exceptional performance that far surpasses that of humans in certain tasks, but to build such a network, a huge amount of data is first required. These networks are used in a wide range of applications, such as extracting knowledge from a large amount of data, learning from past actions, and creating the optimal solution (the optimization problem). A new technique for pre-training natural language processing (NLP) software ver.9.1by using transformers called Bidirectional Encoder Representations (BERT) builds on recent research in pre-training contextual representations, including Semi-Supervised Sequence Learning, Generative Pre-Training, ELMo (Embeddings from Language Models), which is a method for obtaining distributed representations that consider context, and ULMFit (Universal Language Model Fine-Tuning). BERT is a method that can address the issue of the need for large amounts of data, which is inherent in large-scale models, by using pre-learning with unlabeled data. An optimization problem involves “finding a solution that maximizes or minimizes an objective function under given constraints”. In recent years, machine learning approaches that consider pattern recognition as an optimization problem have become popular. This pattern recognition is an operation that associates patterns observed as spatial and temporal changes in signals with classes to which they belong. It involves identifying and retrieving predetermined features and rules from data; however, the features and rules here are not logical information, but are found in images, sounds, etc. Therefore, pattern recognition is generally conducted by supervised learning. Based on a new theory that deals with the process by which the immune system learns from past infection experiences, the clonal selection of immune cells can be viewed as a learning rule of reinforcement learning.

Public Perception of ChatGPT and Transfer Learning for Tweets Sentiment Analysis Using Wolfram Mathematica

Understanding public opinion on ChatGPT is crucial for recognizing its strengths and areas of concern. By utilizing natural language processing (NLP), this study delves into tweets regarding ChatGPT to determine temporal patterns, content features, and topic modeling and perform a sentiment analysis. Analyzing a dataset of 500,000 tweets, our research shifts from conventional data science tools like Python and R to exploit Wolfram Mathematica’s robust capabilities. Additionally, with the aim of solving the problem of ignoring semantic information in the LDA model feature extraction, a synergistic methodology entwining LDA, GloVe embeddings, and K-Nearest Neighbors (KNN) clustering is proposed to categorize topics within ChatGPT-related tweets. This comprehensive strategy ensures semantic, syntactic, and topical congruence within classified groups by utilizing the strengths of probabilistic modeling, semantic embeddings, and similarity-based clustering. While built-in sentiment classifiers often fall short in accuracy, we introduce four transfer learning techniques from the Wolfram Neural Net Repository to address this gap. Two of these techniques involve transferring static word embeddings, “GloVe” and “ConceptNet”, which are further processed using an LSTM layer. The remaining techniques center on fine-tuning pre-trained models using scantily annotated data; one refines embeddings from language models (ELMo), while the other fine-tunes bidirectional encoder representations from transformers (BERT). Our experiments on the dataset underscore the effectiveness of the four methods for the sentiment analysis of tweets. This investigation augments our comprehension of user sentiment towards ChatGPT and emphasizes the continued significance of exploration in this domain. Furthermore, this work serves as a pivotal reference for scholars who are accustomed to using Wolfram Mathematica in other research domains, aiding their efforts in text analytics on social media platforms.

Effectiveness of ELMo embeddings, and semantic models in predicting review helpfulness

Online product reviews (OPR) are a commonly used medium for consumers to communicate their experiences with products during online shopping. Previous studies have investigated the helpfulness of OPRs using frequency-based, linguistic, meta-data, readability, and reviewer attributes. In this study, we explored the impact of robust contextual word embeddings, topic, and language models in predicting the helpfulness of OPRs. In addition, the wrapper-based feature selection technique is employed to select effective subsets from each type of features. Five feature generation techniques including word2vec, FastText, Global Vectors for Word Representation (GloVe), Latent Dirichlet Allocation (LDA), and Embeddings from Language Models (ELMo), were employed. The proposed framework is evaluated on two Amazon datasets (Video games and Health & personal care). The results showed that the ELMo model outperformed the six standard baselines, including the fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model. In addition, ELMo achieved Mean Square Error (MSE) of 0.0887 and 0.0786 respectively on two datasets and MSE of 0.0791 and 0.0708 with the wrapper method. This results in the reduction of 1.43% and 1.63% in MSE as compared to the fine-tuned BERT model on respective datasets. However, the LDA model has a comparable performance with the fine-tuned BERT model but outperforms the other five baselines. The proposed framework demonstrated good generalization abilities by uncovering important factors of product reviews and can be evaluated on other voting platforms.

Shahmukhi named entity recognition by using contextualized word embeddings

Named Entity Recognition (NER) is an imperative Natural Language Processing (NLP) task which intents to identify and classify predefined named entities in a given span of text. For many Western and Asian languages, NER is a systematically premeditated and established task, however, a little work has been done for Shahmukhi. This paper presents Shahmukhi NER with four key contributions. First, a Bi-directional Long-Short Term Memory (BiLSTM) network based NER model has been developed by incorporating various features including character and word embeddings and Part of Speech (POS) tagging. Second, transfer learning has been employed by training context-free Word2Vec and contextualized Embeddings from Language Models (ELMo) word representations. The word representations have been trained using a Shahmukhi corpus of 14.9 million words. Third, we prepared a cleaner version of an existing Shahmukhi NER corpus by performing Unicode normalization and tokenization errands. The corpus has been deduplicated and results are reported on an unseen evaluation set which produced valid results. Fourth, we have studied the impact of two annotation schemes; Inside-Outside (IO) and Inside-Outside-Beginning (IOB) for Shahmukhi. Transfer learning was quite helpful to enhance the performance of NER models especially ELMo embeddings significantly improved the results by prompting contextualized embedding vectors. This is the first study to use character embeddings, POS tagging and transfer learning for Shahmukhi named entity recognition. The IO scheme based model achieved an accuracy of 98.60% with an f-score of 83.75. The IOB scheme based model performed with an accuracy of 98.43% and an f-score of 75.55. These scores are quite promising for an under-resourced morphologically-rich language.

An exploration of the causal factors making an online course content popular & engaging

An education system consists of imparting knowledge, assessing the learners, collecting feedback, analysing it, and taking proper measures to improve it. All these measures ultimately improve learner satisfaction which gets reflected in the learner performance and makes the course popular. Due to the corona pandemic, educational institutes and universities have shifted their activities online. In such a scenario, it is challenging to identify a course's popularity and the factors that make it popular. In this work, the authors have performed aspect-based sentiment analysis of learner feedback to identify the essential aspects of creating a popular course. The traditional methods for analysing feedback for sentiment analysis require manual intervention and are tedious. The authors have proposed a framework that identifies learner reviews' aspect level sentiment polarity and selected the significant aspects impacting a course's popularity. Experimental work is conducted over a large-scale real-world education dataset containing around 110 K learner comments (referred to as Educational Dataset now onwards) collected from Coursera1 and learner data from academic institution MSIT2. Different word embeddings viz., FastText, Word2Vec (Word2Vector), GloVe (Global representation of Vectors), and user-created embeddings are used for CNN (Convolution Neural Network) and LSTM (Long Short Term Memory) models. These models are further compared with BERT (Bidirectional Encoder Representation from Transformers), EvoMSA (Evolutionary Multilingual Sentiment Analysis) and ELMo (Embeddings from Language Model) classifiers to find the method that works best for identifying the aspects that impact the popularity of a course. Based on the experimental results over the collected Dataset, BERT based model provides the best results with an accuracy of 94.27%

Effective method for making Chinese word vector dynamic

Word vector is an important tool for natural language processing (NLP) tasks such as text classification. However, existing static language models such as Word2vec cannot solve the polysemy problem, leading to a decline in text classification performance. To solve this problem, this paper proposes a method for making Chinese word vector dynamic (MCWVD). The part of speech (POS) is used to solve the ambiguity problem caused by different POS. The POS structure graph is constructed and the syntactic structure information of POS features is extracted by GCN (Graph Convolutional Network). POS vector and word vector are concatenated into PW (POS-Word) vector. Parametric matrix is added to improve the fusion effect of POS and word features. Multilayer attention is used to distinguish the importance of different features and further update the vector expression of word vectors about the current context. Experiments on Chinese datasets THUCNews and SogouNews show that MCWVD effectively improves the accuracy of text classification and achieves better performance than CoVe (Context Vectors) and ELMo (Embeddings from Language Models). MCWVD also achieves similar performance to BERT and GPT-1 (Generative Pre-Training), but with a much lower computational cost and only 4% of BERT parameters.

Void Detection inside Duct of Prestressed Concrete Bridges Based on Deep Support Vector Data Description

The tendon that is inserted into the duct is a crucial component of prestressed concrete (PSC) bridges and, when exposed to air, can quickly corrode, and cause structural collapse. It can interpret the signal measured by non-destructive testing (NDT) to determine the condition (normal or void) inside the duct. However, it requires the use of expensive NDT equipment such as ultrasonic waves or the hiring of experts. In this paper, we proposed an impact–echo (IE) method based on deep support vector data description (Deep SVDD) for economical void detection inside a duct. Because the pattern of IE changes for various reasons such as difference of specimen or bridge, supervised learning is not suitable. Deep SVDD is classified as normal and defective, which is a broad distribution as a hypersphere that encloses a multi-dimensional feature space for normal data represented by an autoencoder. Here, an autoencoder was developed based on the ELMo (embeddings from language model)-like structure to obtain an effective representation for IE. In the experiment, we evaluated the performance of the IE data measured in different specimens. Thus, our proposed model showed an accuracy of about 77.84% which is an improvement of up to about 47% compared to the supervised learning approach.

Machine Learning Driven Mental Stress Detection on Reddit Posts Using Natural Language Processing

People’s mental conditions are often reflected in their social media activity due to the internet's anonymity. Psychiatric issues are often detected through such activities and can be addressed in their early stages, potentially preventing the consequences of unattended mental disorders like depression and anxiety. In this paper, the authors have implemented machine learning models and used various embedding techniques to classify posts from the famous social media blog site Reddit as stressful and non-stressful. The dataset used contains user posts that can be analyzed to detect patterns in the social media activity of those diagnosed with mental disorders. This paper uses different NLP (Natural Language Processing) tools such as ELMo (Embeddings from Language Models) word embeddings, BERT (Bidirectional Encoder Representations from Transformers) tokenizers, and BoW (Bag of Words) approach to create word/sentence data that can be fed to machine learning models. The results of each method have been discussed. The results achieved a top F1 score of 0.76, a Precision score of 0.71, and a Recall of 0.74 using only the preprocessed texts and machine learning algorithms to classify the posts. The results achieved by this paper are significant and have the potential to be applied in real-world scenarios to analyze mental stress among social media users. Although this paper focuses on data from Reddit, the techniques used can be transferred to similar social media platforms and could help solve the growing mental health crisis.

How to detect propaganda from social media? Exploitation of semantic and fine-tuned language models.

Online propaganda is a mechanism to influence the opinions of social media users. It is a growing menace to public health, democratic institutions, and public society. The present study proposes a propaganda detection framework as a binary classification model based on a news repository. Several feature models are explored to develop a robust model such as part-of-speech, LIWC, word uni-gram, Embeddings from Language Models (ELMo), FastText, word2vec, latent semantic analysis (LSA), and char tri-gram feature models. Moreover, fine-tuning of the BERT is also performed. Three oversampling methods are investigated to handle the imbalance status of the Qprop dataset. SMOTE Edited Nearest Neighbors (ENN) presented the best results. The fine-tuning of BERT revealed that the BERT-320 sequence length is the best model. As a standalone model, the char tri-gram presented superior performance as compared to other features. The robust performance is observed against the combination of char tri-gram + BERT and char tri-gram + word2vec and they outperformed the two state-of-the-art baselines. In contrast to prior approaches, the addition of feature selection further improves the performance and achieved more than 97.60% recall, f1-score, and AUC on the dev and test part of the dataset. The findings of the present study can be used to organize news articles for various public news websites.

Detecting autism from picture book narratives using deep neural utterance embeddings.

Deficits in the ability to use language in social contexts, including storytelling skills, are observed across the autism spectrum. Development in machine-learning approaches may contribute to clinical psychology and psychiatry, given its potential to support decisions concerning the diagnosis and treatment of psychiatric conditions and disorders. To evaluate the usefulness of deep neural networks for detecting autism spectrum disorder (ASD) from textual utterances, specifically from narrations produced by individuals with ASD. We examined two text encoders: Embeddings from Language Models (ELMo) and Universal Sentence Encoder (USE), and three classification algorithms: XGBoost, support vector machines, and dense neural network layer. We aimed to classify 25 participants with ASD and 25 participants with typical development (TD) based on their narrations produced during the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) picture book task. The results of computational approaches were compared with the results of standardized testing and classifications made by two psychiatrists (raters). The raters were asked to read utterances produced by a participant (without an examiner's statements and additional information) and assign a participant to one of the two groups: ASD or with typical development (TD). The computer-based models had higher sensitivity, specificity, positive predictive values and negative predictive values than the raters, and lower than the two standardized instruments: ADOS-2 and Social Communication Questionnaire (SCQ). Our findings lay the groundwork for future studies involving deep neural network-based text representation models as tools for augmenting the ASD diagnosis or screening. Both ELMo and USE text encoders provided promising specificities, sensitivities, positive predictive values and negative predictive values. Our results indicate the usefulness of page-level embeddings for utterance representation in ADOS-2 picture book task. What is already known on this subject Deficits in the use of language in social contexts, and narrative ability in particular, are observed across the autism spectrum. Most research on narrative skills has applied hand-coding methods. Hitherto, machine-learning methods were used mostly for image recognition problems and data from screening questionnaires for ASD classification. Detection of mental and developmental disorders from textual input is an emerging field for machine and deep-learning methods. What this paper adds to existing knowledge This study explored the ability of several types of deep neural network-based text representation models to detect ASD. Both ELMo and USE provided the most promising values of specificity, sensitivity, positive predictive values and negative predictive values. What are the potential or actual clinical implications of this work? Competitive accuracy, repeatability, speed and ease of operation are all advantages of computerized methods. They allow for objective and quantitative assessment of narrative ability and complex language skills. Deep neural network-based text representation models could in the future support clinicians and augment the decision-making process related to ASD diagnosis, screening and intervention planning.

Embedding arabic questions by feature-level fusion of word representations for questions classification: It is worth doing?

Questions classification remains one of the most critical phases of Question Answering Systems. It aims to reduce the answers search space by assigning a predefined class label to each question. Recently, contextualized word representations methods based on deep learning approach have achieved state of the art performance in various fields of Natural Language Processing. However, few works have applied these representations to classify Arabic questions. In this research, we propose an Arabic question classification method based on a sentence transformers-based representation. Besides, we investigate the fusion of various word representations including Bidirectional Encoder Representations from Transformers (BERT), Embeddings from Language Models (ELMo), and word embeddings enriched by subwords information (W2V). Our contribution is threefold. First, our method handles out of vocabulary words. Second, we apply the BERT representation to extract the most valuable features from words and then construct better question representation. Third, we study the impact of fusing various word embeddings on Arabic question classification. To evaluate the proposed models, we perform stratified 5-folds cross-validation on a dataset containing 3173 questions labeled with Arabic and Li & Roth taxonomies. The experimental results show that all our models surpassed previous works related to Arabic question classification task. In the Arabic taxonomy case, we scored the maximum accuracy of 94.20% with our AraBERT-based model. As for Li & Roth taxonomy, the model based on the concatenation of AraBERT and W2V scored the highest overall accuracy of 93.51%.

A Context-Aware Neural Embedding for Function-Level Vulnerability Detection

Exploitable vulnerabilities in software systems are major security concerns. To date, machine learning (ML) based solutions have been proposed to automate and accelerate the detection of vulnerabilities. Most ML techniques aim to isolate a unit of source code, be it a line or a function, as being vulnerable. We argue that a code segment is vulnerable if it exists in certain semantic contexts, such as the control flow and data flow; therefore, it is important for the detection to be context aware. In this paper, we evaluate the performance of mainstream word embedding techniques in the scenario of software vulnerability detection. Based on the evaluation, we propose a supervised framework leveraging pre-trained context-aware embeddings from language models (ELMo) to capture deep contextual representations, further summarized by a bidirectional long short-term memory (Bi-LSTM) layer for learning long-range code dependency. The framework takes directly a source code function as an input and produces corresponding function embeddings, which can be treated as feature sets for conventional ML classifiers. Experimental results showed that the proposed framework yielded the best performance in its downstream detection tasks. Using the feature representations generated by our framework, random forest and support vector machine outperformed four baseline systems on our data sets, demonstrating that the framework incorporated with ELMo can effectively capture the vulnerable data flow patterns and facilitate the vulnerability detection task.

Measuring Quality of Wikipedia Articles by Feature Fusion‐based Stack Learning

AbstractOnline open‐source knowledge repository such as Wikipedia has become an increasingly important source for users to access knowledge. However, due to its large volume, it is challenging to evaluate Wikipedia article quality manually. To fill this gap, we propose a novel approach named “feature fusion‐based stack learning” to assess the quality of Wikipedia articles. Pre‐trained language models including BERT (Bidirectional Encoder Representations from Transformers) and ELMo (Embeddings from Language Models) are applied to extract semantic information in Wikipedia content. The feature fusion framework consisting of semantic and statistical features is built and fed into an out‐of‐sample (OOS) stacking model, which includes both machine learning and deep learning models. We compare the performance of proposed model with some existing models with different metrics extensively, and conduct ablation studies to prove the effectiveness of our framework and OOS stacking. Generally, the experiment shows that our method is much better than state‐of‐the‐art models.

Detecting formal thought disorder by deep contextualized word representations

Computational linguistics has enabled the introduction of objective tools that measure some of the symptoms of schizophrenia, including the coherence of speech associated with formal thought disorder (FTD). Our goal was to investigate whether neural network based utterance embeddings are more accurate in detecting FTD than models based on individual indicators. The present research used a comprehensive Embeddings from Language Models (ELMo) approach to represent interviews with patients suffering from schizophrenia (N=35) and with healthy people (N=35). We compared its results to the approach described by Bedi et al. (2015), referred to here as the coherence model. Evaluations were also performed by a clinician using the Scale for the Assessment of Thought, Language and Communication (TLC). Using all six TLC questions the ELMo obtained an accuracy of 80% in distinguishing patients from healthy people. Previously used coherence models were less accurate at 70%. The classifying clinician was accurate 74% of the time. Our analysis shows that both ELMo and TLC are sensitive to the symptoms of disorganization in patients. In this study methods using text representations from language models were more accurate than those based solely on the assessment of FTD, and can be used as measures of disordered language that complement human clinical ratings.

A Study on Standardization of Security Evaluation Information for Chemical Processes Based on Deep Learning

Hazard and operability analysis (HAZOP) is one of the most commonly used hazard analysis methods in the petrochemical industry. The large amount of unstructured data in HAZOP reports has generated an information explosion which has led to a pressing need for technologies that can simplify the use of this information. In order to solve the problem that massive data are difficult to reuse and share, in this study, we propose a new deep learning framework for Chinese HAZOP documents to perform a named entity recognition (NER) task, aiming at the characteristics of HAZOP documents, such as polysemy, multi-entity nesting, and long-distance text. Specifically, the preprocessed data are input into an embeddings from language models (ELMo) and a double convolutional neural network (DCNN) model to extract rich character features. Meanwhile, a bidirectional long short-term memory (BiLSTM) network is used to extract long-distance semantic information. Finally, the results are decoded by a conditional random field (CRF), and then output. Experiments were carried out using the HAZOP report of a coal seam indirect liquefaction project. The experimental results for the proposed model showed that the accuracy rate of the optimal results reached 90.83, the recall rate reached 92.46, and the F-value reached the highest 91.76%, which was significantly improved as compared with other models.

Ternary Sentiment Classification of Airline Passengers’ Twitter Text Based on BERT

Traditional word vector generation model cannot solve the polysemy problem of word representation in Twitter text, so, a method is presented here, firstly, by using Bidirectional Encoder Representations from Transformers (BERT), the semantic feature vector of the text can be obtained, and then, the feature vector is inputted into the Softmax classifier to implement the sentiment classification of Twitter text. The experimental datasets are source from passengers’ Twitter comments of USA six airlines, the sentiment classification model based on Embeddings from Language Model (ELMo) as the experimental control group. The experimental results indicate that the proposed model is advantage over the experimental control group by using F1-score as the evaluation index.

Arabic duplicate questions detection based on contextual representation, class label matching, and structured self attention

Question Answering Systems (QAS) are rising solutions providing exact and precise answers to natural questions. Duplicate Question Detection (DQD), which aims to reuse previous answers, has shown its ability to improve user experience and reduce significantly the response time. However, few Arabic QAS integrate solutions able to detect duplicate questions in their workflow. In this paper, we build a DQD method based on contextual word representation, question classification and forward/backward structured self attention. First, we extract contextual word representation Embeddings from Language Models (ELMo) to map questions into a vector space. Next, we train two models to classify question embedding according to two taxonomies: Hamza et al. and Li & Roth. Then, we introduce a class label matching step to filter out questions that have different class labels. Finally, we propose a Bidirectional Attention Bidirectional LSTM (BiAttention BiLSTM) model that focuses only on keywords to predict whether a question pair is a duplicate or not. We also apply a data augmentation strategy based on symmetry, reflexivity, and transitivity relations to improve the generalization of our model. Various experimentations are performed to evaluate the impact of question classification and pre-processing step on DQD model. The obtained results show that our model achieves good performances as compared to the baseline results.