Microsoft Research Paraphrase Corpus Research Articles

Automated Marking System for Essay Questions

The stress of marking assessment scripts of many candidates often results in fatigue that could lead to low productivity and reduced consistency. In most cases, candidates use words, phrases and sentences that are synonyms or related in meaning to those stated in the marking scheme, however, examiners rely solely on the exact words specified in the marking scheme. This often leads to inconsistent grading and in most cases, candidates are disadvantaged. This study seeks to address these inconsistencies during assessment by evaluating the marked answer scripts and the marking scheme of Introduction to File Processing (CSC 221) from the Department of Computer Science, University of Uyo, Nigeria. These were collected and used with the Microsoft Research Paraphrase (MSRP) corpus. After preprocessing the datasets, they were subjected to Logistic Regression (LR), a machine learning technique where the semantic similarity of the answers of the candidates was measured in relation to the marking scheme of the examiner using the MSRP corpus model earlier trained on the Term Frequency-Inverse Document Frequency (TF-IDF) vectorization. Results of the experiment show a strong correlation coefficient of 0.89 and a Mean Relative Error (MRE) of 0.59 compared with the scores awarded by the human marker (examiner). Analysis of the error indicates that block marks were assigned to answers in the marking scheme while the automated marking system breaks the block marks into chunks based on phrases both in the marking scheme and the candidates’ answers. It also shows that some semantically related words were ignored by the examiner.

Effective privacy preserving model based on adversarial CNN with IBOA in the social IoT systems for CEC

SummaryIn the present scenario, the social Internet of Things (IoT) is one of the emerging technologies in combination with collaborative edge computing (CEC). The CEC solves the issue of storage and computing with the help of deep learning models that make full usage of the edge‐computing abilities. The robustness of the deep learning models is ineffective, and the edge devices in the CEC are threatened by the malicious attacks. Therefore, a new data protection framework is proposed in this manuscript to avoid the security crisis and privacy leakage of CEC. A new adversarial sample generation model is introduced in this manuscript on the basis of intensive butterfly optimization algorithm (IBOA) that effectively reduces the time complexity of the framework to linear. Here, IBOA is chosen because an extra intensive exploitation step is included that guides the proposed framework to converge quickly towards global optimum and to avoid local optima trap. Then, the adversarial training concept is incorporated with the convolutional neural network (CNN) model for sentence similarity analysis. On the other hand, the anti‐dropout layers are combined with the adversarial CNN model to reduce the overfitting concern. The proposed framework, adversarial CNN with IBOA, obtained higher results with f1‐score of 88.70%, accuracy of 89.40%, correct rate of 88.95%, recall of 89.48%, and precision of 89.41% on the Microsoft Research Paraphrase Corpus (MSRP) dataset. The obtained results are superiorly better than the conventional deep learning models.

Efficient reinforcement learning-based method for plagiarism detection boosted by a population-based algorithm for pretraining weights

Plagiarism detection (PD) in natural language processing involves locating similar words in two distinct sources. The paper introduces a new approach to plagiarism detection utilizing bidirectional encoder representations from transformers (BERT)-generated embedding, an enhanced artificial bee colony (ABC) optimization algorithm for pre-training, and a training process based on reinforcement learning (RL). The BERT model can be incorporated into a subsequent task and meticulously refined to function as a model, enabling it to apprehend a variety of linguistic characteristics. Imbalanced classification is one of the fundamental obstacles to PD. To handle this predicament, we present a novel methodology utilizing RL, in which the problem is framed as a series of sequential decisions in which an agent receives a reward at each level for classifying a received instance. To address the disparity between classes, it is determined that the majority class will receive a lower reward than the minority class. We also focus on the training stage, which often utilizes gradient-based learning techniques like backpropagation (BP), leading to certain drawbacks such as sensitivity to initialization. In our proposed model, we utilize a mutual learning-based ABC (ML-ABC) approach that adjusts the food source with the most beneficial results for the candidate by considering a mutual learning factor that incorporates the initial weight. We evaluated the efficacy of our novel approach by contrasting its results with those of population-based techniques using three standard datasets, namely Stanford Natural Language Inference (SNLI), Microsoft Research Paraphrase Corpus (MSRP), and Semantic Evaluation Database (SemEval2014). Our model attained excellent results that outperformed state-of-the-art models. Optimal values for important parameters, including reward function are identified for the model based on experiments on the study dataset. Ablation studies that exclude the proposed ML-ABC and reinforcement learning from the model confirm the independent positive incremental impact of these components on model performance.

Sentence-CROBI: A Simple Cross-Bi-Encoder-Based Neural Network Architecture for Paraphrase Identification

Since the rise of Transformer networks and large language models, cross-encoders have become the dominant architecture for various Natural Language Processing tasks. When dealing with sentence pairs, they can exploit the relationships between those pairs. On the other hand, bi-encoders can obtain a vector given a single sentence and are used in tasks such as textual similarity or information retrieval due to their low computational cost; however, their performance is inferior to that of cross-encoders. In this paper, we present Sentence-CROBI, an architecture that combines cross-encoders and bi-encoders to obtain a global representation of sentence pairs. We evaluated the proposed architecture in the paraphrase identification task using the Microsoft Research Paraphrase Corpus, the Quora Question Pairs dataset, and the PAWS-Wiki dataset. Our model obtains competitive results compared with the state-of-the-art by using model ensembles and a simple model configuration. These results demonstrate that a simple architecture that combines sentence pair and single-sentence representations without using complex pre-training or fine-tuning algorithms is a viable alternative for sentence pair tasks.

Automated Paraphrase Quality Assessment Using Language Models and Transfer Learning

Learning to paraphrase supports both writing ability and reading comprehension, particularly for less skilled learners. As such, educational tools that integrate automated evaluations of paraphrases can be used to provide timely feedback to enhance learner paraphrasing skills more efficiently and effectively. Paraphrase identification is a popular NLP classification task that involves establishing whether two sentences share a similar meaning. Paraphrase quality assessment is a slightly more complex task, in which pairs of sentences are evaluated in-depth across multiple dimensions. In this study, we focus on four dimensions: lexical, syntactical, semantic, and overall quality. Our study introduces and evaluates various machine learning models using handcrafted features combined with Extra Trees, Siamese neural networks using BiLSTM RNNs, and pretrained BERT-based models, together with transfer learning from a larger general paraphrase corpus, to estimate the quality of paraphrases across the four dimensions. Two datasets are considered for the tasks involving paraphrase quality: ULPC (User Language Paraphrase Corpus) containing 1998 paraphrases and a smaller dataset with 115 paraphrases based on children’s inputs. The paraphrase identification dataset used for the transfer learning task is the MSRP dataset (Microsoft Research Paraphrase Corpus) containing 5801 paraphrases. On the ULPC dataset, our BERT model improves upon the previous baseline by at least 0.1 in F1-score across the four dimensions. When using fine-tuning from ULPC for the children dataset, both the BERT and Siamese neural network models improve upon their original scores by at least 0.11 F1-score. The results of these experiments suggest that transfer learning using generic paraphrase identification datasets can be successful, while at the same time obtaining comparable results in fewer epochs.

Detección de paráfrasis basada en la energía, entropía y temperatura textual

espanolLa parafrasis es la reformulacion de un texto utilizando un vocabulario distinto para plasmar la idea original con nuestras propias palabras. En esta investigacion se presenta un metodo para la deteccion de parafrasis incorporando los conceptos de Entropia y Temperatura Textual a un modelo previo que centro su contribucion en la implementacion de las redes neuronales recurrentes de Hopfield para generar una medida de distancia llamada Energia Textual. Utilizando la Entropia y la Temperatura se genero un Contexto de Afinidad Libre, basandose en el Model Ising, lo que permitio medir la distribucion semantica entre pares de oraciones. Este modelo fue evaluado utilizando el recurso Microsoft Research Paraphrase Corpus, permitiendo superar los resultados del modelo anterior y logrando identificar mas de la mitad de la parafrasis de la muestra analizada. EnglishParaphrases are the reformulation of a text using different vocabulary to capture the original idea in our own words. In this research, a method for paraphrase detection is presented. We incorporate two variables, Entropy and Textual Temperature, into a previous model which implemented a Hopfield Network to generate a distance measure called Textual Energy. A Context of Free Affinity was generated using Entropy and Temperature based on the Ising Model, which allowed us to measure the semantic distribution between pairs of sentences. Our model was evaluated using Microsoft's Research Paraphrase Corpus improving the results of the previous model and was able to identify more than half of the paraphrases presented in the analyzed sample.

Intelligent recognition of semantic relationships based on antonymy

Since computing semantic similarity tends to simulate the thinking process of humans, semantic dissimilarity must play a part in this process. In this paper, we present a new approach for semantic similarity measuring by taking consideration of dissimilarity into the process of computation. Specifically, the proposed measures explore the potential antonymy in the hierarchical structure of WordNet to represent the dissimilarity between concepts and then combine the dissimilarity with the results of existing methods to achieve semantic similarity results. The relation between parameters and the correlation value is discussed in detail. The proposed model is then applied to different text granularity levels to validate the correctness on similarity measurement. Experimental results show that the proposed approach not only achieves high correlation value against human ratings but also has effective improvement to existing path-distance based methods on the word similarity level, in the meanwhile effectively correct existing sentence similarity method in some cases in Microsoft Research Paraphrase Corpus and SemEval-2014 date set.

Information mining and similarity computation for semi- / un-structured sentences from the social data

In recent years, with the development of the social Internet of Things (IoT), all kinds of data accumulated on the network. These data, which contain a lot of social information and opinions. However, these data are rarely fully analyzed, which is a major obstacle to the intelligent development of the social IoT. In this paper, we propose a sentence similarity analysis model to analyze the similarity in people’s opinions on hot topics in social media and news pages. Most of these data are unstructured or semi-structured sentences, so the accuracy of sentence similarity analysis largely determines the model’s performance. For the purpose of improving accuracy, we propose a novel method of sentence similarity computation to extract the syntactic and semantic information of the semi-structured and unstructured sentences. We mainly consider the subjects, predicates and objects of sentence pairs and use Stanford Parser to classify the dependency relation triples to calculate the syntactic and semantic similarity between two sentences. Finally, we verify the performance of the model with the Microsoft Research Paraphrase Corpus (MRPC), which consists of 4076 pairs of training sentences and 1725 pairs of test sentences, and most of the data came from the news of social data. Extensive simulations demonstrate that our method outperforms other state-of-the-art methods regarding the correlation coefficient and the mean deviation.

Paraphrase Identification with Lexical, Syntactic and Sentential Encodings

Paraphrase identification has been one of the major topics in Natural Language Processing (NLP). However, how to interpret a diversity of contexts such as lexical and semantic information within a sentence as relevant features is still an open problem. This paper addresses the problem and presents an approach for leveraging contextual features with a neural-based learning model. Our Lexical, Syntactic, and Sentential Encodings (LSSE) learning model incorporates Relational Graph Convolutional Networks (R-GCNs) to make use of different features from local contexts, i.e., word encoding, position encoding, and full dependency structures. By utilizing the hidden states obtained by the R-GCNs as well as lexical and sentential encodings by Bidirectional Encoder Representations from Transformers (BERT), our model learns the contextual similarity between sentences effectively. The experimental results by using the two benchmark datasets, Microsoft Research Paraphrase Corpus (MRPC) and Quora Question Pairs (QQP) show that the improvement compared with the baseline, BERT sentential encodings model, was 1.7% F1-score on MRPC and 1.0% F1-score on QQP. Moreover, we verified that the combination of position encoding and syntactic features contributes to performance improvement.

Corpus-Based Paraphrase Detection Experiments and Review

Paraphrase detection is important for a number of applications, including plagiarism detection, authorship attribution, question answering, text summarization, text mining in general, etc. In this paper, we give a performance overview of various types of corpus-based models, especially deep learning (DL) models, with the task of paraphrase detection. We report the results of eight models (LSI, TF-IDF, Word2Vec, Doc2Vec, GloVe, FastText, ELMO, and USE) evaluated on three different public available corpora: Microsoft Research Paraphrase Corpus, Clough and Stevenson and Webis Crowd Paraphrase Corpus 2011. Through a great number of experiments, we decided on the most appropriate approaches for text pre-processing: hyper-parameters, sub-model selection—where they exist (e.g., Skipgram vs. CBOW), distance measures, and semantic similarity/paraphrase detection threshold. Our findings and those of other researchers who have used deep learning models show that DL models are very competitive with traditional state-of-the-art approaches and have potential that should be further developed.

Twice fine‐tuning deep neural networks for paraphrase identification

In this Letter, the authors introduce a novel approach to learn representations for sentence-level paraphrase identification (PI) using BERT and ten natural language processing tasks. Their method trains an unsupervised model called BERT with two different tasks to detect whether two sentences are in paraphrase relation or not. Unlike conventional BERT, which fine tunes the target task such as PI to pre-trained BERT, twice fine-tuning deep neural networks first fine tune each task (e.g. general language understanding evaluation tasks, question answering, and paraphrase adversaries from word scrambling task) and second fine tune target PI task. As a result, the multi-fine-tuned BERT model outperformed the fine-tuned model only with Microsoft Research Paraphrase Corpus (MRPC), which is paraphrase data, except for one case of Stanford Sentiment Treebank - 2 (SST-2). Multi-task fine-tuning is a simple idea but experimentally powerful. Experiments show that fine-tuning just PI tasks to the BERT already gives enough performance, but additionally, fine-tuning similar tasks can affect performance (3.4% point absolute improvement) and be competitive with the state-of-the-art systems.

Learning short-text semantic similarity with word embeddings and external knowledge sources

We present a novel method based on interdependent representations of short texts for determining their degree of semantic similarity. The method represents each short text as two dense vectors: the former is built using the word-to-word similarity based on pre-trained word vectors, the latter is built using the word-to-word similarity based on external sources of knowledge. We also developed a preprocessing algorithm that chains coreferential named entities together and performs word segmentation to preserve the meaning of phrasal verbs and idioms. We evaluated the proposed method on three popular datasets, namely Microsoft Research Paraphrase Corpus, STS2015 and P4PIN, and obtained state-of-the-art results on all three without using prior knowledge of natural language, e.g., part-of-speech tags or parse tree, which indicates the interdependent representations of short text pairs are effective and efficient for semantic textual similarity tasks.

Annotation of subtitle paraphrases using a new web tool

This paper analyzes the manual annotation effort carried out to produce Opusparcus, the Open Subtitles Paraphrase Corpus for six European languages. Within the scope of the project, a new web-based annotation tool was created. We discuss the design choices behind the tool as well as the setup of the annotation task. We also evaluate the annotations obtained. Two independent annotators needed to decide to what extent two sentences approximately meant the same thing. The sentences originate from subtitles from movies and TV shows, which constitutes an interesting genre of mostly colloquial language. Inter-annotator agreement was found to be on par with a well-known previous paraphrase resource from the news domain, the Microsoft Research Paraphrase Corpus (MSRPC). Our annotation tool is open source. The tool can be used for closed projects with restricted access and controlled user authentification as well as open crowdsourced projects, in which anyone can participate and user identification takes place based on IP addresses.

A hybrid approach for paraphrase identification based on knowledge-enriched semantic heuristics

In this paper, we propose a hybrid approach for sentence paraphrase identification. The proposal addresses the problem of evaluating sentence-to-sentence semantic similarity when the sentences contain a set of named-entities. The essence of the proposal is to distinguish the computation of the semantic similarity of named-entity tokens from the rest of the sentence text. More specifically, this is based on the integration of word semantic similarity derived from WordNet taxonomic relations, and named-entity semantic relatedness inferred from Wikipedia entity co-occurrences and underpinned by Normalized Google Distance. In addition, the WordNet similarity measure is enriched with word part-of-speech (PoS) conversion aided with a Categorial Variation database (CatVar), which enhances the lexico-semantics of words. We validated our hybrid approach using two different datasets; Microsoft Research Paraphrase Corpus (MSRPC) and TREC-9 Question Variants. In our empirical evaluation, we showed that our system outperforms baselines and most of the related state-of-the-art systems for paraphrase detection. We also conducted a misidentification analysis to disclose the primary sources of our system errors.

Paraphrase Extraction using fuzzy hierarchical clustering

Paraphrase Extraction involves the discovery of equivalent text segments from large corpora and finds application in tasks such as multi-document summarization and document clustering. Semantic similarity identification is a challenging problem which is further compounded by the large size of the corpus. In this paper a two-stage approach which involves clustering followed by Paraphrase Recognition has been proposed for extraction of sentence-level paraphrases from text collections. In order to handle the ambiguity and inherent variability of natural language a fuzzy hierarchical clustering approach which combines agglomeration based on verbs and division on nouns has been used. Sentences within each resultant cluster are then processed by a machine-learning based Paraphrase Recognizer to discover the paraphrases. The two-stage approach has been applied on the Microsoft Research Paraphrase Corpus and a subset of the Microsoft Research Video Description Corpus. The performance has been evaluated against an existing k-means clustering approach as well as cosine-similarity technique and Fuzzy C-Means clustering and the two-stage system has consistently demonstrated better performance.

Evaluation and classification of syntax usage in determining short-text semantic similarity

This paper outlines and categorizes ways of using syntactic information in a number of algorithms for determining the semantic similarity of short texts. We consider the use of word order information, part-of-speech tagging, parsing and semantic role labeling. We analyze and evaluate the effects of syntax usage on algorithm performance by utilizing the results of a paraphrase detection test on the Microsoft Research Paraphrase Corpus. We also propose a new classification of algorithms based on their applicability to languages with scarce natural language processing tools.

Feature Analysis for Paraphrase Recognition and Textual Entailment

Paraphrase recognition is the task of Natural Language Processing of detecting if an expression restated as another expression contains the same information. Textual Entailment recognition, while being similar to paraphrase recognition, is a task that consists in finding out if a given text can be observed as a consequence of another text fragment, sometimes considering only part of the original meaning, or adding some inferences based on common sense. Tra- ditionally, for solving this problem, several lexical, syntactic and semantic based techniques are used. In this work, we seek to use the less resources as possi- ble, while being effective. For this, we perform a feature analysis for performing Paraphrase Recognition and recognizing Textual Entailment experimenting with the combination of several Natural Language Processing techniques like word overlapping, syntactic analysis, and elimination of stop words. Particularly, we explore using the syntactic n-grams technique combined with some auxiliary ap- proaches such as stemming, synonym detection, similarity measures and linear interpolation. We measure and compare the performance of our system by using the Microsoft Research Paraphrase Corpus, and the RTE-3 test set for Paraphras- ing and Textual Entailment, respectively. Syntactic n-grams produce good results for Paraphrase Recognition. As far as we know, syntactic n-grams had not been used for this task. For Textual Entailment, our best results were obtained by using a simple word overlapping algorithm based on stemming and elimination of stop words.

A Study on Paraphrase Recognition Using Radial Basis Function Neural Network

Paraphrasing is the restatement of a given text using alternate words. Recognition of paraphrases is vital in applications such as question answering, information extraction, and multi-document summarization. Lexical, syntactic, and semantic features of text can be used either individually or in combinations for recognizing paraphrases. Several machine-learning classifiers such as support vector machines (SVM), nearest neighbour method, and decision trees have been used for paraphrase recognition with SVM recognizers being the most popular ones. This paper explores the applicability of neural networks for paraphrase recognition. A radial basis function neural network (RBFNN) has been designed and implemented for recognizing paraphrases. Experiments have been carried out on the Microsoft research paraphrase corpus. From the results of the experiments, it has been observed that the RBFNN recognizer consistently outperforms the SVM recognizer with respect to accuracy and that the best performance was achieved when a combination of lexical, syntactic, and semantic features were used.

Paraphrase Recognition using Neural Network Classification

Paraphrasing refers to conveying the same content in several ways. The successful recognition of paraphrases is crucial to various natural language processing tasks such as Information Extraction, Document Summarization, Question Answering etc. Several techniques have been employed for paraphrase recognition using lexical, syntactic and semantic features. Many of these systems have been tested on the MicroSoft Research Paraphrase Corpus. But the performance of these systems has scope for further improvement. Since neural network architectures model the human brain structure which excels at natural language processing tasks, this paper presents a neural network classifier for recognizing paraphrases. A combination of lexical, syntactic and semantic features has been used to train a Back Propagation network. The system can be utilized for detecting similar sentences in applications such as Question Answering and detection of plagiarized content.

Identification of Sentence-to-Sentence Relations Using a Textual Entailer

We show in this article how an approach developed for the task of recognizing textual entailment relations can be extended to identify paraphrase and elaboration relations. Entailment is a unidirectional relation between two sentences in which one sentence logically infers the other. There seems to be a close relation between entailment and two other sentence-to-sentence relations: elaboration and paraphrase. This close relation is discussed to theoretically justify the newly derived approaches. The proposed approaches use lexical, syntactic, and shallow negation handling. The proposed approaches offer significantly better results than several baselines. When compared to other paraphrase and elaboration approaches they produce similar or better results. We report results on several data sets: the Microsoft Research Paraphrase corpus, a benchmark for evaluating approaches to paraphrase identification, and a data set collected from high-school students' interactions with an intelligent tutoring system iSTART, which includes both paraphrase and elaboration utterances.